Search in sources :

Example 1 with StreamFile

use of com.hedera.mirror.common.domain.StreamFile in project hedera-mirror-node by hashgraph.

the class MirrorDateRangePropertiesProcessor method getLastStreamFile.

/**
 * Gets the latest stream file for downloader based on startDate in MirrorProperties, the startDateAdjustment and
 * last valid downloaded stream file.
 *
 * @param streamType What type of stream to retrieve
 * @return The latest stream file from the database or a dummy stream file if it calculated a different effective
 * start date
 */
public <T extends StreamFile> Optional<T> getLastStreamFile(StreamType streamType) {
    Instant startDate = mirrorProperties.getStartDate();
    Optional<T> streamFile = findLatest(streamType);
    Instant lastFileInstant = streamFile.map(StreamFile::getConsensusStart).map(nanos -> Instant.ofEpochSecond(0, nanos)).orElse(null);
    Instant effectiveStartDate = STARTUP_TIME;
    boolean hasStreamFile = lastFileInstant != null;
    if (startDate != null) {
        effectiveStartDate = max(startDate, hasStreamFile ? lastFileInstant : Instant.EPOCH);
    } else if (hasStreamFile) {
        effectiveStartDate = lastFileInstant;
    } else if (mirrorProperties.getNetwork() == MirrorProperties.HederaNetwork.DEMO) {
        // Demo network contains only data in the past, so don't default to now
        effectiveStartDate = Instant.EPOCH;
    }
    Instant endDate = mirrorProperties.getEndDate();
    if (startDate != null && startDate.compareTo(endDate) > 0) {
        throw new InvalidConfigurationException(String.format("Date range constraint violation: " + "startDate (%s) > endDate (%s)", startDate, endDate));
    }
    if (effectiveStartDate.compareTo(endDate) > 0) {
        throw new InvalidConfigurationException(String.format("Date range constraint violation for %s downloader: effective startDate (%s) > endDate (%s)", streamType, effectiveStartDate, endDate));
    }
    if (!effectiveStartDate.equals(lastFileInstant)) {
        Instant verifyHashAfter = mirrorProperties.getVerifyHashAfter();
        if (verifyHashAfter == null || verifyHashAfter.isBefore(effectiveStartDate)) {
            mirrorProperties.setVerifyHashAfter(effectiveStartDate);
            log.debug("Set verifyHashAfter to {}", effectiveStartDate);
        }
        String filename = StreamFilename.getFilename(streamType, DATA, effectiveStartDate);
        T effectiveStreamFile = (T) ReflectUtils.newInstance(streamType.getStreamFileClass());
        effectiveStreamFile.setConsensusStart(DomainUtils.convertToNanosMax(effectiveStartDate));
        effectiveStreamFile.setName(filename);
        effectiveStreamFile.setIndex(streamFile.map(StreamFile::getIndex).orElse(null));
        streamFile = Optional.of(effectiveStreamFile);
    }
    log.info("{}: downloader will download files in time range ({}, {}]", streamType, effectiveStartDate, mirrorProperties.getEndDate());
    return streamFile;
}
Also used : DownloaderProperties(com.hedera.mirror.importer.downloader.DownloaderProperties) InvalidConfigurationException(com.hedera.mirror.importer.exception.InvalidConfigurationException) RequiredArgsConstructor(lombok.RequiredArgsConstructor) Value(lombok.Value) StreamFilename(com.hedera.mirror.importer.domain.StreamFilename) Duration(java.time.Duration) Map(java.util.Map) Named(javax.inject.Named) Utility(com.hedera.mirror.importer.util.Utility) DomainUtils(com.hedera.mirror.common.util.DomainUtils) ObjectUtils.max(org.apache.commons.lang3.ObjectUtils.max) EventFileRepository(com.hedera.mirror.importer.repository.EventFileRepository) ReflectUtils(org.springframework.cglib.core.ReflectUtils) DATA(com.hedera.mirror.importer.domain.StreamFilename.FileType.DATA) StreamType(com.hedera.mirror.common.domain.StreamType) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) StreamFile(com.hedera.mirror.common.domain.StreamFile) Instant(java.time.Instant) AccountBalanceFileRepository(com.hedera.mirror.importer.repository.AccountBalanceFileRepository) List(java.util.List) MirrorProperties(com.hedera.mirror.importer.MirrorProperties) RecordFileRepository(com.hedera.mirror.importer.repository.RecordFileRepository) Log4j2(lombok.extern.log4j.Log4j2) Optional(java.util.Optional) StreamFileRepository(com.hedera.mirror.importer.repository.StreamFileRepository) Instant(java.time.Instant) StreamFile(com.hedera.mirror.common.domain.StreamFile) InvalidConfigurationException(com.hedera.mirror.importer.exception.InvalidConfigurationException)

Example 2 with StreamFile

use of com.hedera.mirror.common.domain.StreamFile in project hedera-mirror-node by hashgraph.

the class MirrorDateRangePropertiesProcessor method newDateRangeFilter.

private DateRangeFilter newDateRangeFilter(StreamType streamType) {
    DownloaderProperties downloaderProperties = getDownloaderProperties(streamType);
    if (!downloaderProperties.isEnabled()) {
        return DateRangeFilter.empty();
    }
    Instant startDate = mirrorProperties.getStartDate();
    Instant endDate = mirrorProperties.getEndDate();
    Instant lastFileInstant = findLatest(streamType).map(StreamFile::getConsensusStart).map(nanos -> Instant.ofEpochSecond(0, nanos)).orElse(null);
    Instant filterStartDate = lastFileInstant;
    if (startDate != null && startDate.compareTo(endDate) > 0) {
        throw new InvalidConfigurationException(String.format("Date range constraint violation: " + "startDate (%s) > endDate (%s)", startDate, endDate));
    }
    if (startDate != null) {
        filterStartDate = max(startDate, lastFileInstant);
    } else {
        if (mirrorProperties.getNetwork() != MirrorProperties.HederaNetwork.DEMO && lastFileInstant == null) {
            filterStartDate = STARTUP_TIME;
        }
    }
    DateRangeFilter filter = new DateRangeFilter(filterStartDate, endDate);
    log.info("{}: parser will parse items in the range [{}, {}]", downloaderProperties.getStreamType(), filter.getStartAsInstant(), filter.getEndAsInstant());
    return filter;
}
Also used : DownloaderProperties(com.hedera.mirror.importer.downloader.DownloaderProperties) InvalidConfigurationException(com.hedera.mirror.importer.exception.InvalidConfigurationException) RequiredArgsConstructor(lombok.RequiredArgsConstructor) Value(lombok.Value) StreamFilename(com.hedera.mirror.importer.domain.StreamFilename) Duration(java.time.Duration) Map(java.util.Map) Named(javax.inject.Named) Utility(com.hedera.mirror.importer.util.Utility) DomainUtils(com.hedera.mirror.common.util.DomainUtils) ObjectUtils.max(org.apache.commons.lang3.ObjectUtils.max) EventFileRepository(com.hedera.mirror.importer.repository.EventFileRepository) ReflectUtils(org.springframework.cglib.core.ReflectUtils) DATA(com.hedera.mirror.importer.domain.StreamFilename.FileType.DATA) StreamType(com.hedera.mirror.common.domain.StreamType) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) StreamFile(com.hedera.mirror.common.domain.StreamFile) Instant(java.time.Instant) AccountBalanceFileRepository(com.hedera.mirror.importer.repository.AccountBalanceFileRepository) List(java.util.List) MirrorProperties(com.hedera.mirror.importer.MirrorProperties) RecordFileRepository(com.hedera.mirror.importer.repository.RecordFileRepository) Log4j2(lombok.extern.log4j.Log4j2) Optional(java.util.Optional) StreamFileRepository(com.hedera.mirror.importer.repository.StreamFileRepository) DownloaderProperties(com.hedera.mirror.importer.downloader.DownloaderProperties) Instant(java.time.Instant) StreamFile(com.hedera.mirror.common.domain.StreamFile) InvalidConfigurationException(com.hedera.mirror.importer.exception.InvalidConfigurationException)

Example 3 with StreamFile

use of com.hedera.mirror.common.domain.StreamFile in project hedera-mirror-node by hashgraph.

the class AbstractDownloaderTest method verifyStreamFiles.

protected void verifyStreamFiles(List<String> files) {
    ArgumentCaptor<StreamFile> captor = ArgumentCaptor.forClass(StreamFile.class);
    AtomicLong index = new AtomicLong(firstIndex);
    verify(streamFileNotifier, times(files.size())).verified(captor.capture());
    assertThat(captor.getAllValues()).allMatch(s -> files.contains(s.getName())).allMatch(s -> s.getIndex() == null || s.getIndex() == index.getAndIncrement()).allMatch(s -> downloaderProperties.isPersistBytes() ^ (s.getBytes() == null));
}
Also used : EntityId(com.hedera.mirror.common.domain.entity.EntityId) AddressBook(com.hedera.mirror.common.domain.addressbook.AddressBook) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) AddressBookEntry(com.hedera.mirror.common.domain.addressbook.AddressBookEntry) EntityType(com.hedera.mirror.common.domain.entity.EntityType) TransactionType(com.hedera.mirror.common.domain.transaction.TransactionType) MetricsExecutionInterceptor(com.hedera.mirror.importer.config.MetricsExecutionInterceptor) BlobStoreContext(org.jclouds.blobstore.BlobStoreContext) SignatureFileReaderV5(com.hedera.mirror.importer.reader.signature.SignatureFileReaderV5) Pair(org.apache.commons.lang3.tuple.Pair) ExtendWith(org.junit.jupiter.api.extension.ExtendWith) BeforeAll(org.junit.jupiter.api.BeforeAll) SignatureFileReaderV2(com.hedera.mirror.importer.reader.signature.SignatureFileReaderV2) Duration(java.time.Duration) RecordFile(com.hedera.mirror.common.domain.transaction.RecordFile) S3Proxy(org.gaul.s3proxy.S3Proxy) URI(java.net.URI) TestUtils(com.hedera.mirror.importer.TestUtils) Mockito.doReturn(org.mockito.Mockito.doReturn) Path(java.nio.file.Path) MethodSource(org.junit.jupiter.params.provider.MethodSource) MockitoExtension(org.mockito.junit.jupiter.MockitoExtension) CompositeSignatureFileReader(com.hedera.mirror.importer.reader.signature.CompositeSignatureFileReader) S3AsyncClient(software.amazon.awssdk.services.s3.S3AsyncClient) SIGNATURE(com.hedera.mirror.importer.domain.StreamFilename.FileType.SIGNATURE) Predicate(java.util.function.Predicate) ReflectUtils(org.springframework.cglib.core.ReflectUtils) CloudStorageConfiguration(com.hedera.mirror.importer.config.CloudStorageConfiguration) StreamType(com.hedera.mirror.common.domain.StreamType) AddressBookServiceImpl(com.hedera.mirror.importer.addressbook.AddressBookServiceImpl) Set(java.util.Set) Instant(java.time.Instant) Arguments(org.junit.jupiter.params.provider.Arguments) Collectors(java.util.stream.Collectors) FileData(com.hedera.mirror.common.domain.file.FileData) Test(org.junit.jupiter.api.Test) List(java.util.List) Stream(java.util.stream.Stream) TempDir(org.junit.jupiter.api.io.TempDir) Optional(java.util.Optional) CsvSource(org.junit.jupiter.params.provider.CsvSource) Mock(org.mockito.Mock) AbstractLifeCycle(org.gaul.shaded.org.eclipse.jetty.util.component.AbstractLifeCycle) AddressBookService(com.hedera.mirror.importer.addressbook.AddressBookService) ArgumentCaptor(org.mockito.ArgumentCaptor) StreamFilename(com.hedera.mirror.importer.domain.StreamFilename) FileCopier(com.hedera.mirror.importer.FileCopier) LoggingMeterRegistry(io.micrometer.core.instrument.logging.LoggingMeterRegistry) MirrorDateRangePropertiesProcessor(com.hedera.mirror.importer.config.MirrorDateRangePropertiesProcessor) DomainUtils(com.hedera.mirror.common.util.DomainUtils) Properties(java.util.Properties) Files(java.nio.file.Files) DATA(com.hedera.mirror.importer.domain.StreamFilename.FileType.DATA) SignatureFileReader(com.hedera.mirror.importer.reader.signature.SignatureFileReader) FileOutputStream(java.io.FileOutputStream) Mockito.times(org.mockito.Mockito.times) IOException(java.io.IOException) FileUtils(org.apache.commons.io.FileUtils) StreamFile(com.hedera.mirror.common.domain.StreamFile) File(java.io.File) Mockito.verify(org.mockito.Mockito.verify) ContextBuilder(org.jclouds.ContextBuilder) DisplayName(org.junit.jupiter.api.DisplayName) ResourceUtils(org.springframework.util.ResourceUtils) AtomicLong(java.util.concurrent.atomic.AtomicLong) Mockito(org.mockito.Mockito) AfterEach(org.junit.jupiter.api.AfterEach) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MirrorProperties(com.hedera.mirror.importer.MirrorProperties) MeterRegistry(io.micrometer.core.instrument.MeterRegistry) Log4j2(lombok.extern.log4j.Log4j2) FileChannel(java.nio.channels.FileChannel) Collections(java.util.Collections) AtomicLong(java.util.concurrent.atomic.AtomicLong) StreamFile(com.hedera.mirror.common.domain.StreamFile)

Example 4 with StreamFile

use of com.hedera.mirror.common.domain.StreamFile in project hedera-mirror-node by hashgraph.

the class AbstractLinkedStreamDownloaderTest method verifyHashChain.

@ParameterizedTest(name = "verifyHashChain {5}")
@CsvSource({ // @formatter:off
"'', '', 1970-01-01T00:00:00Z,        2000-01-01T10:00:00Z, true,  passes if both hashes are empty", // starting stream in middle
"xx, '', 1970-01-01T00:00:00Z,        2000-01-01T10:00:00Z, true,  passes if hash mismatch and expected hash is empty", // bad db state
"'', xx, 1970-01-01T00:00:00Z,        2000-01-01T10:00:00Z, false, fails if hash mismatch and actual hash is empty", "xx, yy, 1970-01-01T00:00:00Z,        2000-01-01T10:00:00Z, false, fails if hash mismatch and hashes are non-empty", "xx, yy, 2000-01-01T10:00:00.000001Z, 2000-01-01T10:00:00Z, true,  passes if hash mismatch but verifyHashAfter is after filename", "xx, yy, 2000-01-01T10:00:00.000001Z, 2000-01-01T10:00:00Z, true,  passes if hash mismatch but verifyHashAfter is same as filename", "xx, yy, 2000-01-01T09:59:59.999999Z, 2000-01-01T10:00:00Z, false, fails if hash mismatch and verifyHashAfter is before filename", "xx, xx, 1970-01-01T00:00:00Z,        2000-01-01T10:00:00Z, true,  passes if hashes are equal" // @formatter:on
})
void verifyHashChain(String actualPrevFileHash, String expectedPrevFileHash, Instant verifyHashAfter, Instant fileInstant, Boolean expectedResult, String testName) {
    downloaderProperties.getMirrorProperties().setVerifyHashAfter(verifyHashAfter);
    StreamFile streamFile = (StreamFile) ReflectUtils.newInstance(streamType.getStreamFileClass());
    streamFile.setConsensusStart(DomainUtils.convertToNanosMax(fileInstant));
    streamFile.setName(StreamFilename.getFilename(streamType, StreamFilename.FileType.DATA, fileInstant));
    streamFile.setPreviousHash(actualPrevFileHash);
    assertThat(downloader.verifyHashChain(streamFile, expectedPrevFileHash)).as(testName).isEqualTo(expectedResult);
}
Also used : StreamFile(com.hedera.mirror.common.domain.StreamFile) CsvSource(org.junit.jupiter.params.provider.CsvSource) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 5 with StreamFile

use of com.hedera.mirror.common.domain.StreamFile in project hedera-mirror-node by hashgraph.

the class AbstractStreamFileParserTest method failureShouldRollback.

@Test
void failureShouldRollback() {
    // given
    StreamFile streamFile = getStreamFile();
    mockDbFailure();
    // when
    Assertions.assertThrows(ParserException.class, () -> {
        parser.parse(streamFile);
    });
    // then
    assertParsed(streamFile, false, true);
    assertPostParseStreamFile(streamFile, false);
}
Also used : StreamFile(com.hedera.mirror.common.domain.StreamFile) Test(org.junit.jupiter.api.Test)

Aggregations

StreamFile (com.hedera.mirror.common.domain.StreamFile)11 Test (org.junit.jupiter.api.Test)5 StreamType (com.hedera.mirror.common.domain.StreamType)4 Instant (java.time.Instant)4 DomainUtils (com.hedera.mirror.common.util.DomainUtils)3 MirrorProperties (com.hedera.mirror.importer.MirrorProperties)3 StreamFilename (com.hedera.mirror.importer.domain.StreamFilename)3 DATA (com.hedera.mirror.importer.domain.StreamFilename.FileType.DATA)3 Duration (java.time.Duration)3 List (java.util.List)3 Optional (java.util.Optional)3 Log4j2 (lombok.extern.log4j.Log4j2)3 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)3 ReflectUtils (org.springframework.cglib.core.ReflectUtils)3 DownloaderProperties (com.hedera.mirror.importer.downloader.DownloaderProperties)2 InvalidConfigurationException (com.hedera.mirror.importer.exception.InvalidConfigurationException)2 AccountBalanceFileRepository (com.hedera.mirror.importer.repository.AccountBalanceFileRepository)2 EventFileRepository (com.hedera.mirror.importer.repository.EventFileRepository)2 RecordFileRepository (com.hedera.mirror.importer.repository.RecordFileRepository)2 StreamFileRepository (com.hedera.mirror.importer.repository.StreamFileRepository)2