Search in sources :

Example 1 with NewMediaItem

use of com.google.photos.library.v1.proto.NewMediaItem in project data-transfer-project by google.

the class GoogleVideosImporterTest method descriptionOver1kCharactersShouldNotFail.

@Test
public void descriptionOver1kCharactersShouldNotFail() {
    // Mock creation response
    String videoDescriptionOver1k = "z0pyiaeQ03C7Pfs7qdoWfpR2E4BjqqLvsEL1OdBaQu8PeoI5uca83NJQKOfF4gnAhzvpgtbAPGBUade" + "FH5i4vas067Q67aDg1JA9qnnMEy5TTS7Qrp0MImGAI4aHFINwDrTOlFnyGoOwtQC6LLbWWlM8m224G1C08oEyjxuWicMXIdsJfsQvbE" + "dropW0jOMmO2DTCDCPRKwONGHPpo48pmi8HbtNolrbnU189mhyi4zSK3xmMAgmxQWOMmuNryXWB0Zok8hDxnxIes85Oe853U3jUxdu6" + "wNwkbZcpb97dj3puh6UYO9YFFsU40F2ULzpvPTvAPvDeBeOENpUjuh9YhPQiMbwLqne2AxLgMDgxz473Ho2DosixcWjSmX6JfSxInXh" + "lmXxN6xLJRi2abHeEpbOdvl28xEYpBF73DuZZ7NKPKyKhEcWi7aJVoWp9niBl0Cp4PCOO51ABROXOzE8dcoxf6dU1fhqkcQcuxV1qeK" + "XfYewxh8uZeShaMoey1rwzuux7lnKoHDGVQe1nJwSuTUNE5BgLa3uOSwQ9wG0tuakZF2M2YIMhEF6DUu7mZfN41fwPFleuwzO76C6eD" + "inP3xlNJzhsQjQtL0ITCf2oL6LgqLNxzHIRpY41d1Puxzyx2wWJ7DJy3UnMlylyEwhNkd8EuuyXYCs6GIzUXkvHRQZjN99ED6gkmnHS" + "SIW0QHBWOb4jHSYpK52OVMIsLkwRll8zNWci7rRXxFeMw0s0sFcIZthajvP7PMA361bNUDQe4vVhsxF1AQufm0D2SYGpA4zH8LOsacl" + "QPP2vKFFED90jUvbqkhesYYGvrvSq0t12LoMTFqkckRbxj7tODIUco9FFf9U5MQV40q6jgrKup19BSR9NUI58Y0GpI5ZqPgSaNhoJ5V" + "vsPhjrywUo6s9oOnolihQYq6lXZzwhESS8diG34oFLEwq9msSsrRtUSjgH50mNGogOlgEtbaFlMgXstzOWtUk2CwFEHZ9Y2qv123456" + "7890";
    final VideoModel videoModel = new VideoModel(VIDEO_TITLE, VIDEO_URI, videoDescriptionOver1k, MP4_MEDIA_TYPE, VIDEO_ID, null, false);
    String uploadToken = "token";
    NewMediaItem newMediaItemResult = googleVideosImporter.buildMediaItem(videoModel, uploadToken);
    assertFalse("Expected the length of the description to be truncated to 1000 chars.", (newMediaItemResult.getDescription().length() > 1000));
    assertTrue("Expected a truncated description to terminate with \"...\"", newMediaItemResult.getDescription().endsWith("..."));
}
Also used : NewMediaItem(com.google.photos.library.v1.proto.NewMediaItem) VideoModel(org.datatransferproject.types.common.models.videos.VideoModel) Test(org.junit.Test)

Example 2 with NewMediaItem

use of com.google.photos.library.v1.proto.NewMediaItem in project data-transfer-project by google.

the class GoogleVideosImporter method buildMediaItem.

@VisibleForTesting
NewMediaItem buildMediaItem(VideoModel inputVideo, String uploadToken) {
    NewMediaItem newMediaItem;
    String videoDescription = inputVideo.getDescription();
    if (Strings.isNullOrEmpty(videoDescription)) {
        newMediaItem = NewMediaItemFactory.createNewMediaItem(uploadToken);
    } else {
        if (videoDescription.length() > 1000) {
            videoDescription = videoDescription.substring(0, 997) + "...";
        }
        newMediaItem = NewMediaItemFactory.createNewMediaItem(uploadToken, videoDescription);
    }
    return newMediaItem;
}
Also used : NewMediaItem(com.google.photos.library.v1.proto.NewMediaItem) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Example 3 with NewMediaItem

use of com.google.photos.library.v1.proto.NewMediaItem in project data-transfer-project by google.

the class GoogleVideosImporter method importVideoBatch.

long importVideoBatch(List<VideoModel> batchedVideos, PhotosLibraryClient client, IdempotentImportExecutor executor) throws Exception {
    final ArrayList<NewMediaItem> mediaItems = new ArrayList<>();
    final HashMap<String, VideoModel> uploadTokenToDataId = new HashMap<>();
    final HashMap<String, Long> uploadTokenToLength = new HashMap<>();
    // calls of the client to handle the InvalidArgumentException when the user's storage is full.
    try {
        for (VideoModel video : batchedVideos) {
            try {
                Pair<String, Long> pair = uploadMediaItem(video, client);
                final String uploadToken = pair.getLeft();
                mediaItems.add(buildMediaItem(video, uploadToken));
                uploadTokenToDataId.put(uploadToken, video);
                uploadTokenToLength.put(uploadToken, pair.getRight());
            } catch (IOException e) {
                if (e instanceof FileNotFoundException) {
                    // If the video file is no longer available then skip the video. We see this in a small
                    // number of videos where the video has been deleted.
                    monitor.info(() -> String.format("Video resource was missing for id: %s", video.getDataId()), e);
                    continue;
                }
                executor.executeAndSwallowIOExceptions(video.getDataId(), video.getName(), () -> {
                    throw e;
                });
            }
        }
        if (mediaItems.isEmpty()) {
            // Either we were not passed in any videos or we failed upload on all of them.
            return 0L;
        }
        BatchCreateMediaItemsResponse response = client.batchCreateMediaItems(mediaItems);
        final List<NewMediaItemResult> resultsList = response.getNewMediaItemResultsList();
        long bytes = 0L;
        for (NewMediaItemResult result : resultsList) {
            String uploadToken = result.getUploadToken();
            Status status = result.getStatus();
            final VideoModel video = uploadTokenToDataId.get(uploadToken);
            Preconditions.checkNotNull(video);
            final int code = status.getCode();
            if (code == Code.OK_VALUE) {
                executor.executeAndSwallowIOExceptions(video.getDataId(), video.getName(), () -> result.getMediaItem().getId());
                Long length = uploadTokenToLength.get(uploadToken);
                if (length != null) {
                    bytes += length;
                }
            } else {
                executor.executeAndSwallowIOExceptions(video.getDataId(), video.getName(), () -> {
                    throw new IOException(String.format("Video item could not be created. Code: %d Message: %s", code, result.getStatus().getMessage()));
                });
            }
            uploadTokenToDataId.remove(uploadToken);
        }
        if (!uploadTokenToDataId.isEmpty()) {
            for (VideoModel video : uploadTokenToDataId.values()) {
                executor.executeAndSwallowIOExceptions(video.getDataId(), video.getName(), () -> {
                    throw new IOException("Video item was missing from results list.");
                });
            }
        }
        return bytes;
    } catch (InvalidArgumentException e) {
        if (e.getMessage().contains("The remaining storage in the user's account is not enough")) {
            throw new DestinationMemoryFullException("Google destination storage full", e);
        } else {
            throw e;
        }
    }
}
Also used : Status(com.google.rpc.Status) HashMap(java.util.HashMap) DestinationMemoryFullException(org.datatransferproject.spi.transfer.types.DestinationMemoryFullException) ArrayList(java.util.ArrayList) FileNotFoundException(java.io.FileNotFoundException) NewMediaItemResult(com.google.photos.library.v1.proto.NewMediaItemResult) IOException(java.io.IOException) VideoModel(org.datatransferproject.types.common.models.videos.VideoModel) InvalidArgumentException(com.google.api.gax.rpc.InvalidArgumentException) NewMediaItem(com.google.photos.library.v1.proto.NewMediaItem) BatchCreateMediaItemsResponse(com.google.photos.library.v1.proto.BatchCreateMediaItemsResponse)

Aggregations

NewMediaItem (com.google.photos.library.v1.proto.NewMediaItem)3 VideoModel (org.datatransferproject.types.common.models.videos.VideoModel)2 InvalidArgumentException (com.google.api.gax.rpc.InvalidArgumentException)1 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 BatchCreateMediaItemsResponse (com.google.photos.library.v1.proto.BatchCreateMediaItemsResponse)1 NewMediaItemResult (com.google.photos.library.v1.proto.NewMediaItemResult)1 Status (com.google.rpc.Status)1 FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 DestinationMemoryFullException (org.datatransferproject.spi.transfer.types.DestinationMemoryFullException)1 Test (org.junit.Test)1