Search in sources :

Example 1 with IdOnlyResource

use of org.dataportabilityproject.shared.IdOnlyResource in project data-transfer-project by google.

the class GoogleCalendarService method exportCalendars.

private CalendarModelWrapper exportCalendars(Optional<PaginationInformation> pageInfo) throws IOException {
    Calendar.CalendarList.List listRequest = calendarClient.calendarList().list();
    if (pageInfo.isPresent()) {
        listRequest.setPageToken(((StringPaginationToken) pageInfo.get()).getId());
    }
    CalendarList listResult = listRequest.execute();
    List<CalendarModel> calendarModels = new ArrayList<>(listResult.getItems().size());
    List<Resource> resources = new ArrayList<>(listResult.getItems().size());
    for (CalendarListEntry calendarData : listResult.getItems()) {
        CalendarModel model = GoogleCalendarToModelConverter.convertToCalendarModel(calendarData);
        resources.add(new IdOnlyResource(calendarData.getId()));
        calendarModels.add(model);
    }
    PaginationInformation newPageInfo = null;
    if (listResult.getNextPageToken() != null) {
        newPageInfo = new StringPaginationToken(listResult.getNextPageToken());
    }
    return new CalendarModelWrapper(calendarModels, null, new ContinuationInformation(resources, newPageInfo));
}
Also used : ArrayList(java.util.ArrayList) Resource(org.dataportabilityproject.dataModels.Resource) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) CalendarModel(org.dataportabilityproject.dataModels.calendar.CalendarModel) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) CalendarModelWrapper(org.dataportabilityproject.dataModels.calendar.CalendarModelWrapper) CalendarListEntry(com.google.api.services.calendar.model.CalendarListEntry) ContinuationInformation(org.dataportabilityproject.dataModels.ContinuationInformation) CalendarList(com.google.api.services.calendar.model.CalendarList) PaginationInformation(org.dataportabilityproject.dataModels.PaginationInformation) StringPaginationToken(org.dataportabilityproject.shared.StringPaginationToken)

Example 2 with IdOnlyResource

use of org.dataportabilityproject.shared.IdOnlyResource in project data-transfer-project by google.

the class ImapMailHelper method getMessages.

/**
 * Get all messages in an account.
 */
private MailModelWrapper getMessages(String host, String account, String password, Folder parentFolder, boolean fetchMessages, PaginationInformation paginationInformation) throws MessagingException, IOException {
    int foldersSize = 0;
    // Find containers to and be imported
    ImmutableCollection.Builder<MailContainerModel> folders = ImmutableList.builder();
    ImmutableCollection.Builder<Resource> folderIds = ImmutableList.builder();
    log("Calling list for folder: %s", parentFolder.getName());
    Folder[] subFolders = parentFolder.list();
    log("Folder: %s, subFolders: %d", parentFolder.getName(), subFolders.length);
    for (Folder folder : subFolders) {
        // This will tell the framework to create this folder on import
        folders.add(new MailContainerModel(folder.getName(), folder.getFullName()));
        // Content for these resources will be 'fetched' by the framework
        folderIds.add(new IdOnlyResource(folder.getName()));
        foldersSize++;
    }
    log("foldersSize: %d", foldersSize);
    // Get messages in the folder
    ImmutableCollection.Builder<MailMessageModel> resources = ImmutableList.builder();
    log("fetchMessages: %b", fetchMessages);
    PaginationInformation nextPaginationInformation = null;
    if (fetchMessages) {
        parentFolder.open(Folder.READ_ONLY);
        int start = getStart(paginationInformation);
        int end = getEnd(start, parentFolder.getMessageCount());
        if (end < parentFolder.getMessageCount()) {
            // Indicates page to be fetched on next request
            nextPaginationInformation = new IntPaginationToken(end + 1);
        }
        log("Fetching messages for foder: %s, start: %d, end: %d", parentFolder.getFullName(), start, end);
        Message[] messages = parentFolder.getMessages(start, end);
        log("Fetched message for folder: %s, messages: %s", parentFolder.getFullName(), messages.length);
        for (Message message : messages) {
            log("Message, contentType: %s ,size: %s", message.getContentType(), message.getSize());
            ImmutableList<String> folderId = ImmutableList.of(parentFolder.getName());
            resources.add(new MailMessageModel(createRawMessage(message), folderId));
        }
        parentFolder.close(false);
    }
    // TODO: add pagination below
    return new MailModelWrapper(folders.build(), resources.build(), new ContinuationInformation(folderIds.build(), nextPaginationInformation));
}
Also used : IntPaginationToken(org.dataportabilityproject.shared.IntPaginationToken) ImmutableCollection(com.google.common.collect.ImmutableCollection) Message(javax.mail.Message) Resource(org.dataportabilityproject.dataModels.Resource) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) Folder(javax.mail.Folder) ContinuationInformation(org.dataportabilityproject.dataModels.ContinuationInformation) MailContainerModel(org.dataportabilityproject.dataModels.mail.MailContainerModel) MailModelWrapper(org.dataportabilityproject.dataModels.mail.MailModelWrapper) MailMessageModel(org.dataportabilityproject.dataModels.mail.MailMessageModel) PaginationInformation(org.dataportabilityproject.dataModels.PaginationInformation)

Example 3 with IdOnlyResource

use of org.dataportabilityproject.shared.IdOnlyResource in project data-transfer-project by google.

the class GoogleCalendarServiceTest method testExportCalendarFirstSet.

@Test
public void testExportCalendarFirstSet() throws IOException {
    setUpSingleCalendarResponse();
    // Looking at first page, with at least one page after it
    ExportInformation emptyExportInformation = new ExportInformation(Optional.empty(), Optional.empty());
    calendarListResponse.setNextPageToken(NEXT_TOKEN);
    // Run test
    CalendarModelWrapper wrapper = calendarService.export(emptyExportInformation);
    // Check results
    // Verify correct methods were called
    verify(calendarClient).calendarList();
    verify(calendarCalendarList).list();
    verify(calendarListRequest).execute();
    // Check pagination token
    StringPaginationToken paginationToken = (StringPaginationToken) wrapper.getContinuationInformation().getPaginationInformation();
    assertThat(paginationToken.getId()).isEqualTo(NEXT_TOKEN);
    // Check calendars
    Collection<CalendarModel> calendars = wrapper.getCalendars();
    assertThat(calendars.stream().map(CalendarModel::getId).collect(Collectors.toList())).containsExactly(CALENDAR_ID);
    // Check events (should be empty, even though there is an event in the calendar)
    Collection<CalendarEventModel> events = wrapper.getEvents();
    assertThat(events).isEmpty();
    // Should be one event in the resource list
    Collection<? extends Resource> subResources = wrapper.getContinuationInformation().getSubResources();
    assertThat(subResources.stream().map(a -> ((IdOnlyResource) a).getId()).collect(Collectors.toList())).containsExactly(CALENDAR_ID);
}
Also used : CalendarModelWrapper(org.dataportabilityproject.dataModels.calendar.CalendarModelWrapper) Events(com.google.api.services.calendar.model.Events) InMemoryJobDataCache(org.dataportabilityproject.cloud.local.InMemoryJobDataCache) PaginationInformation(org.dataportabilityproject.dataModels.PaginationInformation) JobDataCache(org.dataportabilityproject.cloud.interfaces.JobDataCache) Calendar(com.google.api.services.calendar.Calendar) GoogleStaticObjects(org.dataportabilityproject.serviceProviders.google.GoogleStaticObjects) StringPaginationToken(org.dataportabilityproject.shared.StringPaginationToken) Before(org.junit.Before) Event(com.google.api.services.calendar.model.Event) InOrder(org.mockito.InOrder) Resource(org.dataportabilityproject.dataModels.Resource) ExportInformation(org.dataportabilityproject.dataModels.ExportInformation) CalendarList(com.google.api.services.calendar.model.CalendarList) Collection(java.util.Collection) CalendarModelWrapper(org.dataportabilityproject.dataModels.calendar.CalendarModelWrapper) IOException(java.io.IOException) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) Truth.assertThat(com.google.common.truth.Truth.assertThat) Collectors(java.util.stream.Collectors) Mockito.verify(org.mockito.Mockito.verify) CalendarEventModel(org.dataportabilityproject.dataModels.calendar.CalendarEventModel) CalendarModel(org.dataportabilityproject.dataModels.calendar.CalendarModel) Mockito(org.mockito.Mockito) CalendarListEntry(com.google.api.services.calendar.model.CalendarListEntry) Optional(java.util.Optional) Collections(java.util.Collections) Mockito.mock(org.mockito.Mockito.mock) ExportInformation(org.dataportabilityproject.dataModels.ExportInformation) CalendarModel(org.dataportabilityproject.dataModels.calendar.CalendarModel) CalendarEventModel(org.dataportabilityproject.dataModels.calendar.CalendarEventModel) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) StringPaginationToken(org.dataportabilityproject.shared.StringPaginationToken) Test(org.junit.Test)

Example 4 with IdOnlyResource

use of org.dataportabilityproject.shared.IdOnlyResource in project data-transfer-project by google.

the class SmugMugPhotoService method getAlbums.

private PhotosModelWrapper getAlbums(Optional<PaginationInformation> paginationInformation) throws IOException {
    String albumUri;
    if (paginationInformation.isPresent()) {
        albumUri = ((StringPaginationToken) paginationInformation.get()).getId();
    } else {
        SmugMugResponse<SmugMugUserResponse> userResponse = makeUserRequest(USER_URL);
        albumUri = userResponse.getResponse().getUser().getUris().get("UserAlbums").getUri();
    }
    List<PhotoAlbum> albums = new ArrayList<>();
    List<Resource> resources = new ArrayList<>();
    SmugMugResponse<SmugmugAlbumsResponse> albumResponse = makeAlbumRequest(albumUri);
    for (SmugMugAlbum album : albumResponse.getResponse().getAlbums()) {
        albums.add(new PhotoAlbum(album.getAlbumKey(), album.getTitle(), album.getDescription()));
        resources.add(new IdOnlyResource(album.getAlbumKey()));
    }
    StringPaginationToken pageToken = null;
    if (albumResponse.getResponse().getPageInfo() != null && albumResponse.getResponse().getPageInfo().getNextPage() != null) {
        pageToken = new StringPaginationToken(albumResponse.getResponse().getPageInfo().getNextPage());
    }
    return new PhotosModelWrapper(albums, null, new ContinuationInformation(resources, pageToken));
}
Also used : SmugMugUserResponse(org.dataportabilityproject.serviceProviders.smugmug.model.SmugMugUserResponse) ArrayList(java.util.ArrayList) Resource(org.dataportabilityproject.dataModels.Resource) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) PhotosModelWrapper(org.dataportabilityproject.dataModels.photos.PhotosModelWrapper) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) SmugmugAlbumsResponse(org.dataportabilityproject.serviceProviders.smugmug.model.SmugmugAlbumsResponse) ContinuationInformation(org.dataportabilityproject.dataModels.ContinuationInformation) SmugMugAlbum(org.dataportabilityproject.serviceProviders.smugmug.model.SmugMugAlbum) PhotoAlbum(org.dataportabilityproject.dataModels.photos.PhotoAlbum) StringPaginationToken(org.dataportabilityproject.shared.StringPaginationToken)

Example 5 with IdOnlyResource

use of org.dataportabilityproject.shared.IdOnlyResource in project data-transfer-project by google.

the class MicrosoftCalendarService method getCalendars.

private CalendarModelWrapper getCalendars(Optional<PaginationInformation> pageInfo) throws IOException {
    URL url = new URL("https://outlook.office.com/api/v2.0/me/calendars");
    HttpRequest getRequest = requestFactory.buildGetRequest(new GenericUrl(url));
    getRequest.setParser(new JsonObjectParser(new JacksonFactory()));
    HttpResponse response;
    try {
        response = getRequest.execute();
    } catch (HttpResponseException e) {
        logger.debug("Error fetching content");
        logger.debug("response status code: {}", e.getStatusCode());
        logger.debug("response status message: {}", e.getStatusMessage());
        logger.debug("response headers: {}", e.getHeaders());
        logger.debug("response content: {}", e.getContent());
        e.printStackTrace();
        throw e;
    }
    int statusCode = response.getStatusCode();
    if (statusCode != 200) {
        throw new IOException("Bad status code: " + statusCode + " error: " + response.getStatusMessage());
    }
    // Parse response into model
    OutlookCalendarList data = response.parseAs(OutlookCalendarList.class);
    List<CalendarModel> calendars = new ArrayList<>(data.list.size());
    List<Resource> resources = new ArrayList<>(data.list.size());
    for (OutlookCalendar calendar : data.list) {
        calendars.add(new CalendarModel(calendar.id, calendar.name, null));
        resources.add(new IdOnlyResource(calendar.id));
    }
    return new CalendarModelWrapper(calendars, null, new ContinuationInformation(resources, null));
}
Also used : HttpRequest(com.google.api.client.http.HttpRequest) ArrayList(java.util.ArrayList) Resource(org.dataportabilityproject.dataModels.Resource) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) HttpResponse(com.google.api.client.http.HttpResponse) CalendarModel(org.dataportabilityproject.dataModels.calendar.CalendarModel) HttpResponseException(com.google.api.client.http.HttpResponseException) GenericUrl(com.google.api.client.http.GenericUrl) IOException(java.io.IOException) IdOnlyResource(org.dataportabilityproject.shared.IdOnlyResource) JacksonFactory(com.google.api.client.json.jackson2.JacksonFactory) URL(java.net.URL) CalendarModelWrapper(org.dataportabilityproject.dataModels.calendar.CalendarModelWrapper) ContinuationInformation(org.dataportabilityproject.dataModels.ContinuationInformation) JsonObjectParser(com.google.api.client.json.JsonObjectParser)

Aggregations

IdOnlyResource (org.dataportabilityproject.shared.IdOnlyResource)14 Resource (org.dataportabilityproject.dataModels.Resource)11 ContinuationInformation (org.dataportabilityproject.dataModels.ContinuationInformation)9 ArrayList (java.util.ArrayList)7 StringPaginationToken (org.dataportabilityproject.shared.StringPaginationToken)6 IOException (java.io.IOException)5 ExportInformation (org.dataportabilityproject.dataModels.ExportInformation)5 PaginationInformation (org.dataportabilityproject.dataModels.PaginationInformation)5 CalendarModelWrapper (org.dataportabilityproject.dataModels.calendar.CalendarModelWrapper)5 PhotosModelWrapper (org.dataportabilityproject.dataModels.photos.PhotosModelWrapper)5 PhotoAlbum (org.dataportabilityproject.dataModels.photos.PhotoAlbum)4 Test (org.junit.Test)4 CalendarModel (org.dataportabilityproject.dataModels.calendar.CalendarModel)3 Photoset (com.flickr4java.flickr.photosets.Photoset)2 Photosets (com.flickr4java.flickr.photosets.Photosets)2 CalendarList (com.google.api.services.calendar.model.CalendarList)2 CalendarListEntry (com.google.api.services.calendar.model.CalendarListEntry)2 URL (java.net.URL)2 CalendarEventModel (org.dataportabilityproject.dataModels.calendar.CalendarEventModel)2 TaskListModel (org.dataportabilityproject.dataModels.tasks.TaskListModel)2