Search in sources :

Example 6 with Importer

use of org.datatransferproject.spi.transfer.provider.Importer in project data-transfer-project by google.

the class KoofrTransferExtension method initialize.

@Override
public void initialize(ExtensionContext context) {
    // rather than throwing if called multiple times.
    if (initialized)
        return;
    JobStore jobStore = context.getService(JobStore.class);
    HttpTransport httpTransport = context.getService(HttpTransport.class);
    JsonFactory jsonFactory = context.getService(JsonFactory.class);
    OkHttpClient client = new OkHttpClient.Builder().build();
    ObjectMapper mapper = new ObjectMapper();
    AppCredentials appCredentials;
    try {
        appCredentials = context.getService(AppCredentialStore.class).getAppCredentials("KOOFR_KEY", "KOOFR_SECRET");
    } catch (IOException e) {
        Monitor monitor = context.getMonitor();
        monitor.info(() -> "Unable to retrieve Koofr AppCredentials. Did you set KOOFR_KEY and KOOFR_SECRET?");
        return;
    }
    // Create the KoofrCredentialFactory with the given {@link AppCredentials}.
    KoofrCredentialFactory credentialFactory = new KoofrCredentialFactory(httpTransport, jsonFactory, appCredentials);
    Monitor monitor = context.getMonitor();
    int fileUploadReadTimeout = context.getSetting("koofrFileUploadReadTimeout", 60000);
    int fileUploadWriteTimeout = context.getSetting("koofrFileUploadWriteTimeout", 60000);
    monitor.info(() -> format("Configuring Koofr HTTP file upload client with read timeout %d ms and write timeout %d ms", fileUploadReadTimeout, fileUploadWriteTimeout));
    OkHttpClient fileUploadClient = client.newBuilder().readTimeout(fileUploadReadTimeout, TimeUnit.MILLISECONDS).writeTimeout(fileUploadReadTimeout, TimeUnit.MILLISECONDS).build();
    KoofrClientFactory koofrClientFactory = new KoofrClientFactory(BASE_API_URL, client, fileUploadClient, mapper, monitor, credentialFactory);
    ImmutableMap.Builder<String, Importer> importBuilder = ImmutableMap.builder();
    importBuilder.put(PHOTOS, new KoofrPhotosImporter(koofrClientFactory, monitor, jobStore));
    importBuilder.put(VIDEOS, new KoofrVideosImporter(koofrClientFactory, monitor));
    importerMap = importBuilder.build();
    ImmutableMap.Builder<String, Exporter> exportBuilder = ImmutableMap.builder();
    exportBuilder.put(PHOTOS, new KoofrPhotosExporter(koofrClientFactory, monitor));
    exportBuilder.put(VIDEOS, new KoofrVideosExporter(koofrClientFactory, monitor));
    exporterMap = exportBuilder.build();
    initialized = true;
}
Also used : KoofrPhotosExporter(org.datatransferproject.transfer.koofr.photos.KoofrPhotosExporter) OkHttpClient(okhttp3.OkHttpClient) KoofrClientFactory(org.datatransferproject.transfer.koofr.common.KoofrClientFactory) AppCredentials(org.datatransferproject.types.transfer.auth.AppCredentials) KoofrCredentialFactory(org.datatransferproject.transfer.koofr.common.KoofrCredentialFactory) JsonFactory(com.google.api.client.json.JsonFactory) JobStore(org.datatransferproject.spi.cloud.storage.JobStore) KoofrVideosImporter(org.datatransferproject.transfer.koofr.videos.KoofrVideosImporter) IOException(java.io.IOException) Exporter(org.datatransferproject.spi.transfer.provider.Exporter) KoofrPhotosExporter(org.datatransferproject.transfer.koofr.photos.KoofrPhotosExporter) KoofrVideosExporter(org.datatransferproject.transfer.koofr.videos.KoofrVideosExporter) ImmutableMap(com.google.common.collect.ImmutableMap) HttpTransport(com.google.api.client.http.HttpTransport) KoofrPhotosImporter(org.datatransferproject.transfer.koofr.photos.KoofrPhotosImporter) Monitor(org.datatransferproject.api.launcher.Monitor) KoofrVideosExporter(org.datatransferproject.transfer.koofr.videos.KoofrVideosExporter) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) KoofrVideosImporter(org.datatransferproject.transfer.koofr.videos.KoofrVideosImporter) Importer(org.datatransferproject.spi.transfer.provider.Importer) KoofrPhotosImporter(org.datatransferproject.transfer.koofr.photos.KoofrPhotosImporter)

Example 7 with Importer

use of org.datatransferproject.spi.transfer.provider.Importer in project data-transfer-project by google.

the class GoogleTransferExtension method initialize.

@Override
public void initialize(ExtensionContext context) {
    // times.
    if (initialized)
        return;
    JobStore jobStore = context.getService(JobStore.class);
    HttpTransport httpTransport = context.getService(HttpTransport.class);
    JsonFactory jsonFactory = context.getService(JsonFactory.class);
    AppCredentials appCredentials;
    try {
        appCredentials = context.getService(AppCredentialStore.class).getAppCredentials("GOOGLE_KEY", "GOOGLE_SECRET");
    } catch (IOException e) {
        Monitor monitor = context.getMonitor();
        monitor.info(() -> "Unable to retrieve Google AppCredentials. Did you set GOOGLE_KEY and GOOGLE_SECRET?");
        return;
    }
    Monitor monitor = context.getMonitor();
    // Create the GoogleCredentialFactory with the given {@link AppCredentials}.
    GoogleCredentialFactory credentialFactory = new GoogleCredentialFactory(httpTransport, jsonFactory, appCredentials, monitor);
    ImmutableMap.Builder<String, Importer> importerBuilder = ImmutableMap.builder();
    importerBuilder.put("BLOBS", new DriveImporter(credentialFactory, jobStore, monitor));
    importerBuilder.put("CONTACTS", new GoogleContactsImporter(credentialFactory));
    importerBuilder.put("CALENDAR", new GoogleCalendarImporter(credentialFactory));
    importerBuilder.put("MAIL", new GoogleMailImporter(credentialFactory, monitor));
    importerBuilder.put("TASKS", new GoogleTasksImporter(credentialFactory));
    importerBuilder.put("PHOTOS", new GooglePhotosImporter(credentialFactory, jobStore, jsonFactory, monitor, context.getSetting("googleWritesPerSecond", 1.0)));
    importerBuilder.put("VIDEOS", new GoogleVideosImporter(appCredentials, jobStore, monitor));
    importerMap = importerBuilder.build();
    ImmutableMap.Builder<String, Exporter> exporterBuilder = ImmutableMap.builder();
    exporterBuilder.put("BLOBS", new DriveExporter(credentialFactory, jobStore, monitor));
    exporterBuilder.put("CONTACTS", new GoogleContactsExporter(credentialFactory));
    exporterBuilder.put("CALENDAR", new GoogleCalendarExporter(credentialFactory));
    exporterBuilder.put("MAIL", new GoogleMailExporter(credentialFactory));
    exporterBuilder.put("SOCIAL-POSTS", new GooglePlusExporter(credentialFactory));
    exporterBuilder.put("TASKS", new GoogleTasksExporter(credentialFactory, monitor));
    exporterBuilder.put("PHOTOS", new GooglePhotosExporter(credentialFactory, jobStore, jsonFactory, monitor));
    exporterBuilder.put("VIDEOS", new GoogleVideosExporter(credentialFactory, jsonFactory));
    exporterMap = exporterBuilder.build();
    initialized = true;
}
Also used : GoogleCalendarImporter(org.datatransferproject.datatransfer.google.calendar.GoogleCalendarImporter) GoogleTasksExporter(org.datatransferproject.datatransfer.google.tasks.GoogleTasksExporter) AppCredentials(org.datatransferproject.types.transfer.auth.AppCredentials) GooglePhotosImporter(org.datatransferproject.datatransfer.google.photos.GooglePhotosImporter) JsonFactory(com.google.api.client.json.JsonFactory) GoogleMailImporter(org.datatransferproject.datatransfer.google.mail.GoogleMailImporter) GoogleCredentialFactory(org.datatransferproject.datatransfer.google.common.GoogleCredentialFactory) GoogleCalendarExporter(org.datatransferproject.datatransfer.google.calendar.GoogleCalendarExporter) GoogleVideosExporter(org.datatransferproject.datatransfer.google.videos.GoogleVideosExporter) DriveExporter(org.datatransferproject.datatransfer.google.drive.DriveExporter) GoogleMailExporter(org.datatransferproject.datatransfer.google.mail.GoogleMailExporter) Exporter(org.datatransferproject.spi.transfer.provider.Exporter) GoogleTasksExporter(org.datatransferproject.datatransfer.google.tasks.GoogleTasksExporter) GooglePlusExporter(org.datatransferproject.datatransfer.google.gplus.GooglePlusExporter) GooglePhotosExporter(org.datatransferproject.datatransfer.google.photos.GooglePhotosExporter) GoogleContactsExporter(org.datatransferproject.datatransfer.google.contacts.GoogleContactsExporter) GoogleContactsExporter(org.datatransferproject.datatransfer.google.contacts.GoogleContactsExporter) Monitor(org.datatransferproject.api.launcher.Monitor) GooglePlusExporter(org.datatransferproject.datatransfer.google.gplus.GooglePlusExporter) GoogleVideosImporter(org.datatransferproject.datatransfer.google.videos.GoogleVideosImporter) GooglePhotosImporter(org.datatransferproject.datatransfer.google.photos.GooglePhotosImporter) GoogleContactsImporter(org.datatransferproject.datatransfer.google.contacts.GoogleContactsImporter) Importer(org.datatransferproject.spi.transfer.provider.Importer) GoogleMailImporter(org.datatransferproject.datatransfer.google.mail.GoogleMailImporter) GoogleCalendarImporter(org.datatransferproject.datatransfer.google.calendar.GoogleCalendarImporter) DriveImporter(org.datatransferproject.datatransfer.google.drive.DriveImporter) GoogleVideosImporter(org.datatransferproject.datatransfer.google.videos.GoogleVideosImporter) GoogleTasksImporter(org.datatransferproject.datatransfer.google.tasks.GoogleTasksImporter) GoogleContactsImporter(org.datatransferproject.datatransfer.google.contacts.GoogleContactsImporter) DriveImporter(org.datatransferproject.datatransfer.google.drive.DriveImporter) JobStore(org.datatransferproject.spi.cloud.storage.JobStore) GoogleVideosExporter(org.datatransferproject.datatransfer.google.videos.GoogleVideosExporter) IOException(java.io.IOException) ImmutableMap(com.google.common.collect.ImmutableMap) GooglePhotosExporter(org.datatransferproject.datatransfer.google.photos.GooglePhotosExporter) HttpTransport(com.google.api.client.http.HttpTransport) GoogleCalendarExporter(org.datatransferproject.datatransfer.google.calendar.GoogleCalendarExporter) GoogleTasksImporter(org.datatransferproject.datatransfer.google.tasks.GoogleTasksImporter) DriveExporter(org.datatransferproject.datatransfer.google.drive.DriveExporter) GoogleMailExporter(org.datatransferproject.datatransfer.google.mail.GoogleMailExporter)

Example 8 with Importer

use of org.datatransferproject.spi.transfer.provider.Importer in project data-transfer-project by google.

the class FacebookTransferExtension method initialize.

@Override
public void initialize(ExtensionContext context) {
    if (initialized)
        return;
    AppCredentials appCredentials;
    final Monitor monitor = context.getMonitor();
    try {
        appCredentials = context.getService(AppCredentialStore.class).getAppCredentials("FACEBOOK_KEY", "FACEBOOK_SECRET");
    } catch (IOException e) {
        monitor.info(() -> "Unable to retrieve Facebook AppCredentials. Did you set FACEBOOK_KEY and FACEBOOK_SECRET?", e);
        return;
    }
    ImmutableMap.Builder<String, Importer> importerBuilder = ImmutableMap.builder();
    importerBuilder.put("VIDEOS", new FacebookVideosImporter(appCredentials));
    importerMap = importerBuilder.build();
    ImmutableMap.Builder<String, Exporter> exporterBuilder = ImmutableMap.builder();
    exporterBuilder.put("PHOTOS", new FacebookPhotosExporter(appCredentials, monitor, context.getService(TemporaryPerJobDataStore.class)));
    exporterBuilder.put("VIDEOS", new FacebookVideosExporter(appCredentials, monitor));
    exporterMap = exporterBuilder.build();
    initialized = true;
}
Also used : Monitor(org.datatransferproject.api.launcher.Monitor) AppCredentials(org.datatransferproject.types.transfer.auth.AppCredentials) FacebookPhotosExporter(org.datatransferproject.transfer.facebook.photos.FacebookPhotosExporter) FacebookVideosExporter(org.datatransferproject.transfer.facebook.videos.FacebookVideosExporter) FacebookVideosImporter(org.datatransferproject.transfer.facebook.videos.FacebookVideosImporter) IOException(java.io.IOException) Exporter(org.datatransferproject.spi.transfer.provider.Exporter) FacebookVideosExporter(org.datatransferproject.transfer.facebook.videos.FacebookVideosExporter) FacebookPhotosExporter(org.datatransferproject.transfer.facebook.photos.FacebookPhotosExporter) ImmutableMap(com.google.common.collect.ImmutableMap) Importer(org.datatransferproject.spi.transfer.provider.Importer) FacebookVideosImporter(org.datatransferproject.transfer.facebook.videos.FacebookVideosImporter)

Aggregations

Importer (org.datatransferproject.spi.transfer.provider.Importer)8 ImmutableMap (com.google.common.collect.ImmutableMap)7 Monitor (org.datatransferproject.api.launcher.Monitor)7 IOException (java.io.IOException)5 Exporter (org.datatransferproject.spi.transfer.provider.Exporter)5 AppCredentials (org.datatransferproject.types.transfer.auth.AppCredentials)5 HttpTransport (com.google.api.client.http.HttpTransport)4 JsonFactory (com.google.api.client.json.JsonFactory)4 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)3 OkHttpClient (okhttp3.OkHttpClient)3 TemporaryPerJobDataStore (org.datatransferproject.spi.cloud.storage.TemporaryPerJobDataStore)3 GoogleCredentialFactory (org.datatransferproject.datatransfer.google.common.GoogleCredentialFactory)2 JobStore (org.datatransferproject.spi.cloud.storage.JobStore)2 AuthTestDriver (org.datatransferproject.auth.microsoft.harness.AuthTestDriver)1 BackblazeDataTransferClientFactory (org.datatransferproject.datatransfer.backblaze.common.BackblazeDataTransferClientFactory)1 BackblazePhotosImporter (org.datatransferproject.datatransfer.backblaze.photos.BackblazePhotosImporter)1 BackblazeVideosImporter (org.datatransferproject.datatransfer.backblaze.videos.BackblazeVideosImporter)1 GoogleBloggerImporter (org.datatransferproject.datatransfer.google.blogger.GoogleBloggerImporter)1 GoogleCalendarExporter (org.datatransferproject.datatransfer.google.calendar.GoogleCalendarExporter)1 GoogleCalendarImporter (org.datatransferproject.datatransfer.google.calendar.GoogleCalendarImporter)1