Search in sources :

Example 1 with MoveAfterCopy

use of org.neo4j.com.storecopy.MoveAfterCopy in project neo4j by neo4j.

the class SwitchToSlaveCopyThenBranchTest method shouldNotBranchStoreUnlessWeHaveCopiedDownAReplacement.

@Test
public void shouldNotBranchStoreUnlessWeHaveCopiedDownAReplacement() throws Throwable {
    // Given
    StoreCopyClient storeCopyClient = mock(StoreCopyClient.class);
    doAnswer(invocation -> {
        MoveAfterCopy moveAfterCopy = invocation.getArgumentAt(2, MoveAfterCopy.class);
        moveAfterCopy.move(Stream.empty(), new File(""), new File(""));
        return null;
    }).when(storeCopyClient).copyStore(any(StoreCopyClient.StoreCopyRequester.class), any(CancellationRequest.class), any(MoveAfterCopy.class));
    PageCache pageCacheMock = mock(PageCache.class);
    PagedFile pagedFileMock = mock(PagedFile.class);
    when(pagedFileMock.getLastPageId()).thenReturn(1L);
    when(pageCacheMock.map(any(File.class), anyInt())).thenReturn(pagedFileMock);
    SwitchToSlaveCopyThenBranch switchToSlave = newSwitchToSlaveSpy(pageCacheMock, storeCopyClient);
    URI masterUri = new URI("cluster://localhost?serverId=1");
    URI me = new URI("cluster://localhost?serverId=2");
    CancellationRequest cancellationRequest = CancellationRequest.NEVER_CANCELLED;
    MasterClient masterClient = mock(MasterClient.class);
    when(masterClient.handshake(anyLong(), any(StoreId.class))).thenThrow(new BranchedDataException(""));
    TransactionIdStore transactionIdStore = mock(TransactionIdStore.class);
    when(transactionIdStore.getLastCommittedTransaction()).thenReturn(new TransactionId(42, 42, 42));
    when(transactionIdStore.getLastCommittedTransactionId()).thenReturn(TransactionIdStore.BASE_TX_ID);
    // When
    BranchedDataPolicy branchPolicy = mock(BranchedDataPolicy.class);
    switchToSlave.stopServicesAndHandleBranchedStore(branchPolicy, masterUri, me, cancellationRequest);
    // Then
    InOrder inOrder = Mockito.inOrder(storeCopyClient, branchPolicy);
    inOrder.verify(storeCopyClient).copyStore(any(StoreCopyClient.StoreCopyRequester.class), any(CancellationRequest.class), any(MoveAfterCopy.class));
    inOrder.verify(branchPolicy).handle(new File(""), pageCacheMock, NullLogService.getInstance());
}
Also used : PagedFile(org.neo4j.io.pagecache.PagedFile) TransactionIdStore(org.neo4j.kernel.impl.transaction.log.TransactionIdStore) InOrder(org.mockito.InOrder) MasterClient(org.neo4j.kernel.ha.com.slave.MasterClient) URI(java.net.URI) TransactionId(org.neo4j.kernel.impl.store.TransactionId) StoreId(org.neo4j.kernel.impl.store.StoreId) StoreCopyClient(org.neo4j.com.storecopy.StoreCopyClient) BranchedDataPolicy(org.neo4j.kernel.ha.BranchedDataPolicy) PagedFile(org.neo4j.io.pagecache.PagedFile) File(java.io.File) BranchedDataException(org.neo4j.kernel.ha.BranchedDataException) MoveAfterCopy(org.neo4j.com.storecopy.MoveAfterCopy) CancellationRequest(org.neo4j.helpers.CancellationRequest) PageCache(org.neo4j.io.pagecache.PageCache) Test(org.junit.Test)

Example 2 with MoveAfterCopy

use of org.neo4j.com.storecopy.MoveAfterCopy in project neo4j by neo4j.

the class SwitchToSlaveCopyThenBranch method stopServicesAndHandleBranchedStore.

void stopServicesAndHandleBranchedStore(BranchedDataPolicy branchPolicy, URI masterUri, URI me, CancellationRequest cancellationRequest) throws Throwable {
    MoveAfterCopy moveWithCopyThenBranch = (moves, fromDirectory, toDirectory) -> {
        stopServices();
        msgLog.debug("Branching store: " + storeDir);
        branchPolicy.handle(storeDir, pageCache, logService);
        msgLog.debug("Moving downloaded store from " + fromDirectory + " to " + toDirectory);
        MoveAfterCopy.moveReplaceExisting().move(moves, fromDirectory, toDirectory);
        msgLog.debug("Moved downloaded store from " + fromDirectory + " to " + toDirectory);
    };
    copyStore(masterUri, me, cancellationRequest, moveWithCopyThenBranch);
}
Also used : BranchedDataPolicy(org.neo4j.kernel.ha.BranchedDataPolicy) StoreId(org.neo4j.kernel.impl.store.StoreId) MoveAfterCopy(org.neo4j.com.storecopy.MoveAfterCopy) NeoStoreDataSource(org.neo4j.kernel.NeoStoreDataSource) MasterClientResolver(org.neo4j.kernel.ha.com.slave.MasterClientResolver) TransactionIdStore(org.neo4j.kernel.impl.transaction.log.TransactionIdStore) Monitors(org.neo4j.kernel.monitoring.Monitors) DelegateInvocationHandler(org.neo4j.kernel.ha.DelegateInvocationHandler) StoreUnableToParticipateInClusterException(org.neo4j.kernel.ha.StoreUnableToParticipateInClusterException) LifeSupport(org.neo4j.kernel.lifecycle.LifeSupport) Function(java.util.function.Function) Supplier(java.util.function.Supplier) PullerFactory(org.neo4j.kernel.ha.PullerFactory) MasterClient(org.neo4j.kernel.ha.com.slave.MasterClient) SlaveServer(org.neo4j.kernel.ha.com.slave.SlaveServer) HaIdGeneratorFactory(org.neo4j.kernel.ha.id.HaIdGeneratorFactory) CancellationRequest(org.neo4j.helpers.CancellationRequest) URI(java.net.URI) MismatchingStoreIdException(org.neo4j.kernel.impl.store.MismatchingStoreIdException) BASE_TX_ID(org.neo4j.kernel.impl.transaction.log.TransactionIdStore.BASE_TX_ID) PageCache(org.neo4j.io.pagecache.PageCache) Config(org.neo4j.kernel.configuration.Config) IOException(java.io.IOException) LogService(org.neo4j.kernel.impl.logging.LogService) HaSettings(org.neo4j.kernel.ha.HaSettings) File(java.io.File) UpdatePuller(org.neo4j.kernel.ha.UpdatePuller) ForeignStoreException(org.neo4j.kernel.ha.store.ForeignStoreException) UnableToCopyStoreFromOldMasterException(org.neo4j.kernel.ha.store.UnableToCopyStoreFromOldMasterException) ClusterMemberAvailability(org.neo4j.cluster.member.ClusterMemberAvailability) KernelExtensionFactory(org.neo4j.kernel.extension.KernelExtensionFactory) DependencyResolver(org.neo4j.graphdb.DependencyResolver) Slave(org.neo4j.kernel.ha.com.master.Slave) StoreCopyClient(org.neo4j.com.storecopy.StoreCopyClient) TransactionStats(org.neo4j.kernel.impl.transaction.TransactionStats) RequestContextFactory(org.neo4j.kernel.ha.com.RequestContextFactory) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) Master(org.neo4j.kernel.ha.com.master.Master) MoveAfterCopy(org.neo4j.com.storecopy.MoveAfterCopy)

Example 3 with MoveAfterCopy

use of org.neo4j.com.storecopy.MoveAfterCopy in project neo4j by neo4j.

the class SwitchToSlave method copyStoreFromMaster.

void copyStoreFromMaster(MasterClient masterClient, CancellationRequest cancellationRequest, MoveAfterCopy moveAfterCopy) throws Throwable {
    try {
        userLog.info("Copying store from master");
        StoreCopyClient.StoreCopyRequester requester = new StoreCopyClient.StoreCopyRequester() {

            @Override
            public Response<?> copyStore(StoreWriter writer) {
                return masterClient.copyStore(new RequestContext(0, config.get(ClusterSettings.server_id).toIntegerIndex(), 0, BASE_TX_ID, 0), writer);
            }

            @Override
            public void done() {
            // Nothing to clean up here
            }
        };
        MoveAfterCopy moveAfterCopyWithLogging = (moves, fromDirectory, toDirectory) -> {
            userLog.info("Copied store from master to " + fromDirectory);
            msgLog.info("Starting post copy operation to move store from " + fromDirectory + " to " + storeDir);
            moveAfterCopy.move(moves, fromDirectory, toDirectory);
        };
        storeCopyClient.copyStore(requester, cancellationRequest, moveAfterCopyWithLogging);
        startServicesAgain();
        userLog.info("Finished copying store from master");
    } catch (Throwable t) {
        // Delete store so that we can copy from master without conflicts next time
        cleanStoreDir();
        throw t;
    }
}
Also used : InstanceId(org.neo4j.cluster.InstanceId) StoreId(org.neo4j.kernel.impl.store.StoreId) NeoStoreDataSource(org.neo4j.kernel.NeoStoreDataSource) MasterClientResolver(org.neo4j.kernel.ha.com.slave.MasterClientResolver) Log(org.neo4j.logging.Log) OnlineBackupKernelExtension(org.neo4j.backup.OnlineBackupKernelExtension) DelegateInvocationHandler(org.neo4j.kernel.ha.DelegateInvocationHandler) LifeSupport(org.neo4j.kernel.lifecycle.LifeSupport) RequestContext(org.neo4j.com.RequestContext) MasterClient(org.neo4j.kernel.ha.com.slave.MasterClient) HighAvailabilityModeSwitcher(org.neo4j.kernel.ha.cluster.modeswitch.HighAvailabilityModeSwitcher) TransactionObligationFulfiller(org.neo4j.com.storecopy.TransactionObligationFulfiller) URI(java.net.URI) MismatchingStoreIdException(org.neo4j.kernel.impl.store.MismatchingStoreIdException) StoreUtil(org.neo4j.com.storecopy.StoreUtil) ClusterMembers.hasInstanceId(org.neo4j.kernel.ha.cluster.member.ClusterMembers.hasInstanceId) Response(org.neo4j.com.Response) PageCache(org.neo4j.io.pagecache.PageCache) Iterables.filter(org.neo4j.helpers.collection.Iterables.filter) MissingLogDataException(org.neo4j.kernel.impl.transaction.log.MissingLogDataException) LogService(org.neo4j.kernel.impl.logging.LogService) MILLISECONDS(java.util.concurrent.TimeUnit.MILLISECONDS) TransactionCommittingResponseUnpacker(org.neo4j.com.storecopy.TransactionCommittingResponseUnpacker) UpdatePuller(org.neo4j.kernel.ha.UpdatePuller) UpdatePullerScheduler(org.neo4j.kernel.ha.UpdatePullerScheduler) UnableToCopyStoreFromOldMasterException(org.neo4j.kernel.ha.store.UnableToCopyStoreFromOldMasterException) ClusterMemberAvailability(org.neo4j.cluster.member.ClusterMemberAvailability) Iterables.firstOrNull(org.neo4j.helpers.collection.Iterables.firstOrNull) DependencyResolver(org.neo4j.graphdb.DependencyResolver) RequestContextFactory(org.neo4j.kernel.ha.com.RequestContextFactory) BranchedDataException(org.neo4j.kernel.ha.BranchedDataException) MoveAfterCopy(org.neo4j.com.storecopy.MoveAfterCopy) TransactionIdStore(org.neo4j.kernel.impl.transaction.log.TransactionIdStore) StoreWriter(org.neo4j.com.storecopy.StoreWriter) Monitors(org.neo4j.kernel.monitoring.Monitors) LockSupport.parkNanos(java.util.concurrent.locks.LockSupport.parkNanos) HandshakeResult(org.neo4j.kernel.ha.com.master.HandshakeResult) Function(java.util.function.Function) Supplier(java.util.function.Supplier) PullerFactory(org.neo4j.kernel.ha.PullerFactory) ServerUtil(org.neo4j.com.ServerUtil) TransactionId(org.neo4j.kernel.impl.store.TransactionId) ClusterMembers(org.neo4j.kernel.ha.cluster.member.ClusterMembers) StoreLockerLifecycleAdapter(org.neo4j.kernel.internal.StoreLockerLifecycleAdapter) SlaveServer(org.neo4j.kernel.ha.com.slave.SlaveServer) HaIdGeneratorFactory(org.neo4j.kernel.ha.id.HaIdGeneratorFactory) CancellationRequest(org.neo4j.helpers.CancellationRequest) HighAvailabilityModeSwitcher.getServerId(org.neo4j.kernel.ha.cluster.modeswitch.HighAvailabilityModeSwitcher.getServerId) NeoStores.isStorePresent(org.neo4j.kernel.impl.store.NeoStores.isStorePresent) BASE_TX_ID(org.neo4j.kernel.impl.transaction.log.TransactionIdStore.BASE_TX_ID) Lifecycle(org.neo4j.kernel.lifecycle.Lifecycle) Config(org.neo4j.kernel.configuration.Config) IOException(java.io.IOException) HaSettings(org.neo4j.kernel.ha.HaSettings) DataSourceManager(org.neo4j.kernel.impl.transaction.state.DataSourceManager) File(java.io.File) StoreOutOfDateException(org.neo4j.kernel.ha.StoreOutOfDateException) Server(org.neo4j.com.Server) SlaveImpl(org.neo4j.kernel.ha.com.slave.SlaveImpl) IndexConfigStore(org.neo4j.kernel.impl.index.IndexConfigStore) Clock(java.time.Clock) ClusterSettings(org.neo4j.cluster.ClusterSettings) Slave(org.neo4j.kernel.ha.com.master.Slave) StoreCopyClient(org.neo4j.com.storecopy.StoreCopyClient) TransactionStats(org.neo4j.kernel.impl.transaction.TransactionStats) ClusterMember(org.neo4j.kernel.ha.cluster.member.ClusterMember) FileSystemWatcherService(org.neo4j.kernel.impl.util.watcher.FileSystemWatcherService) Clocks(org.neo4j.time.Clocks) Master(org.neo4j.kernel.ha.com.master.Master) StoreWriter(org.neo4j.com.storecopy.StoreWriter) StoreCopyClient(org.neo4j.com.storecopy.StoreCopyClient) RequestContext(org.neo4j.com.RequestContext) MoveAfterCopy(org.neo4j.com.storecopy.MoveAfterCopy)

Aggregations

File (java.io.File)3 URI (java.net.URI)3 MoveAfterCopy (org.neo4j.com.storecopy.MoveAfterCopy)3 StoreCopyClient (org.neo4j.com.storecopy.StoreCopyClient)3 CancellationRequest (org.neo4j.helpers.CancellationRequest)3 PageCache (org.neo4j.io.pagecache.PageCache)3 IOException (java.io.IOException)2 Function (java.util.function.Function)2 Supplier (java.util.function.Supplier)2 ClusterMemberAvailability (org.neo4j.cluster.member.ClusterMemberAvailability)2 DependencyResolver (org.neo4j.graphdb.DependencyResolver)2 NeoStoreDataSource (org.neo4j.kernel.NeoStoreDataSource)2 Config (org.neo4j.kernel.configuration.Config)2 BranchedDataException (org.neo4j.kernel.ha.BranchedDataException)2 BranchedDataPolicy (org.neo4j.kernel.ha.BranchedDataPolicy)2 DelegateInvocationHandler (org.neo4j.kernel.ha.DelegateInvocationHandler)2 HaSettings (org.neo4j.kernel.ha.HaSettings)2 PullerFactory (org.neo4j.kernel.ha.PullerFactory)2 UpdatePuller (org.neo4j.kernel.ha.UpdatePuller)2 MasterClient (org.neo4j.kernel.ha.com.slave.MasterClient)2