Search in sources :

Example 11 with JsonParserException

use of com.grack.nanojson.JsonParserException in project NewPipeExtractor by TeamNewPipe.

the class YoutubeSuggestionExtractor method suggestionList.

@Override
public List<String> suggestionList(String query) throws IOException, ExtractionException {
    final Downloader dl = NewPipe.getDownloader();
    final List<String> suggestions = new ArrayList<>();
    final String url = "https://suggestqueries.google.com/complete/search" + "?client=" + // "firefox" for JSON, 'toolbar' for xml
    "youtube" + "&jsonp=" + "JP" + "&ds=" + "yt" + "&gl=" + URLEncoder.encode(getExtractorContentCountry().getCountryCode(), UTF_8) + "&q=" + URLEncoder.encode(query, UTF_8);
    final Map<String, List<String>> headers = new HashMap<>();
    addCookieHeader(headers);
    String response = dl.get(url, headers, getExtractorLocalization()).responseBody();
    // trim JSONP part "JP(...)"
    response = response.substring(3, response.length() - 1);
    try {
        JsonArray collection = JsonParser.array().from(response).getArray(1);
        for (Object suggestion : collection) {
            if (!(suggestion instanceof JsonArray))
                continue;
            String suggestionStr = ((JsonArray) suggestion).getString(0);
            if (suggestionStr == null)
                continue;
            suggestions.add(suggestionStr);
        }
        return suggestions;
    } catch (JsonParserException e) {
        throw new ParsingException("Could not parse json response", e);
    }
}
Also used : JsonArray(com.grack.nanojson.JsonArray) JsonParserException(com.grack.nanojson.JsonParserException) ParsingException(org.schabi.newpipe.extractor.exceptions.ParsingException) Downloader(org.schabi.newpipe.extractor.downloader.Downloader)

Example 12 with JsonParserException

use of com.grack.nanojson.JsonParserException in project NewPipe by TeamNewPipe.

the class CheckForNewAppVersion method handleResponse.

private void handleResponse(@NonNull final Response response, @NonNull final NewVersionManager manager, @NonNull final SharedPreferences prefs, @NonNull final App app) {
    try {
        // Store a timestamp which needs to be exceeded,
        // before a new request to the API is made.
        final long newExpiry = manager.coerceExpiry(response.getHeader("expires"));
        prefs.edit().putLong(app.getString(R.string.update_expiry_key), newExpiry).apply();
    } catch (final Exception e) {
        if (DEBUG) {
            Log.w(TAG, "Could not extract and save new expiry date", e);
        }
    }
    // Parse the json from the response.
    try {
        final JsonObject githubStableObject = JsonParser.object().from(response.responseBody()).getObject("flavors").getObject("github").getObject("stable");
        final String versionName = githubStableObject.getString("version");
        final int versionCode = githubStableObject.getInt("version_code");
        final String apkLocationUrl = githubStableObject.getString("apk");
        compareAppVersionAndShowNotification(app, versionName, apkLocationUrl, versionCode);
    } catch (final JsonParserException e) {
        // Do not alarm user and fail silently.
        if (DEBUG) {
            Log.w(TAG, "Could not get NewPipe API: invalid json", e);
        }
    }
}
Also used : JsonParserException(com.grack.nanojson.JsonParserException) JsonObject(com.grack.nanojson.JsonObject) ReCaptchaException(org.schabi.newpipe.extractor.exceptions.ReCaptchaException) JsonParserException(com.grack.nanojson.JsonParserException) IOException(java.io.IOException) CertificateException(java.security.cert.CertificateException) NoSuchAlgorithmException(java.security.NoSuchAlgorithmException) CertificateEncodingException(java.security.cert.CertificateEncodingException)

Example 13 with JsonParserException

use of com.grack.nanojson.JsonParserException in project NewPipe by TeamNewPipe.

the class ServiceHelper method initService.

public static void initService(final Context context, final int serviceId) {
    if (serviceId == ServiceList.PeerTube.getServiceId()) {
        final SharedPreferences sharedPreferences = PreferenceManager.getDefaultSharedPreferences(context);
        final String json = sharedPreferences.getString(context.getString(R.string.peertube_selected_instance_key), null);
        if (null == json) {
            return;
        }
        final JsonObject jsonObject;
        try {
            jsonObject = JsonParser.object().from(json);
        } catch (final JsonParserException e) {
            return;
        }
        final String name = jsonObject.getString("name");
        final String url = jsonObject.getString("url");
        final PeertubeInstance instance = new PeertubeInstance(url, name);
        ServiceList.PeerTube.setInstance(instance);
    }
}
Also used : SharedPreferences(android.content.SharedPreferences) JsonParserException(com.grack.nanojson.JsonParserException) JsonObject(com.grack.nanojson.JsonObject) PeertubeInstance(org.schabi.newpipe.extractor.services.peertube.PeertubeInstance)

Example 14 with JsonParserException

use of com.grack.nanojson.JsonParserException in project NewPipe by TeamNewPipe.

the class TabsJsonHelper method getTabsFromJson.

/**
 * Try to reads the passed JSON and returns the list of tabs if no error were encountered.
 * <p>
 * If the JSON is null or empty, or the list of tabs that it represents is empty, the
 * {@link #getDefaultTabs fallback list} will be returned.
 * <p>
 * Tabs with invalid ids (i.e. not in the {@link Tab.Type} enum) will be ignored.
 *
 * @param tabsJson a JSON string got from {@link #getJsonToSave(List)}.
 * @return a list of {@link Tab tabs}.
 * @throws InvalidJsonException if the JSON string is not valid
 */
public static List<Tab> getTabsFromJson(@Nullable final String tabsJson) throws InvalidJsonException {
    if (tabsJson == null || tabsJson.isEmpty()) {
        return getDefaultTabs();
    }
    final List<Tab> returnTabs = new ArrayList<>();
    final JsonObject outerJsonObject;
    try {
        outerJsonObject = JsonParser.object().from(tabsJson);
        if (!outerJsonObject.has(JSON_TABS_ARRAY_KEY)) {
            throw new InvalidJsonException("JSON doesn't contain \"" + JSON_TABS_ARRAY_KEY + "\" array");
        }
        final JsonArray tabsArray = outerJsonObject.getArray(JSON_TABS_ARRAY_KEY);
        for (final Object o : tabsArray) {
            if (!(o instanceof JsonObject)) {
                continue;
            }
            final Tab tab = Tab.from((JsonObject) o);
            if (tab != null) {
                returnTabs.add(tab);
            }
        }
    } catch (final JsonParserException e) {
        throw new InvalidJsonException(e);
    }
    if (returnTabs.isEmpty()) {
        return getDefaultTabs();
    }
    return returnTabs;
}
Also used : JsonArray(com.grack.nanojson.JsonArray) JsonParserException(com.grack.nanojson.JsonParserException) ArrayList(java.util.ArrayList) JsonObject(com.grack.nanojson.JsonObject) JsonObject(com.grack.nanojson.JsonObject)

Example 15 with JsonParserException

use of com.grack.nanojson.JsonParserException in project Piped-Backend by TeamPiped.

the class DownloaderImpl method executeRequest.

/**
 * Executes a request with HTTP/2.
 */
public Response executeRequest(Request request) throws IOException, ReCaptchaException {
    // TODO: HTTP/3 aka QUIC
    var bytes = request.dataToSend();
    RequestBody body = null;
    if (bytes != null)
        body = RequestBody.create(bytes);
    var builder = new okhttp3.Request.Builder().url(request.url()).method(request.httpMethod(), body).header("User-Agent", Constants.USER_AGENT);
    if (saved_cookie != null && !saved_cookie.hasExpired())
        builder.header("Cookie", saved_cookie.getName() + "=" + saved_cookie.getValue());
    request.headers().forEach((name, values) -> values.forEach(value -> builder.header(name, value)));
    var resp = Constants.h2client.newCall(builder.build()).execute();
    if (resp.code() == 429) {
        synchronized (cookie_lock) {
            if (saved_cookie != null && saved_cookie.hasExpired() || (System.currentTimeMillis() - cookie_received > TimeUnit.MINUTES.toMillis(30)))
                saved_cookie = null;
            String redir_url = String.valueOf(resp.request().url());
            if (saved_cookie == null && redir_url.startsWith("https://www.google.com/sorry")) {
                var formBuilder = new FormBody.Builder();
                String sitekey = null, data_s = null;
                for (Element el : Jsoup.parse(resp.body().string()).selectFirst("form").children()) {
                    String name;
                    if (!(name = el.tagName()).equals("script")) {
                        if (name.equals("input"))
                            formBuilder.add(el.attr("name"), el.attr("value"));
                        else if (name.equals("div") && el.attr("id").equals("recaptcha")) {
                            sitekey = el.attr("data-sitekey");
                            data_s = el.attr("data-s");
                        }
                    }
                }
                if (StringUtils.isEmpty(sitekey) || StringUtils.isEmpty(data_s))
                    throw new ReCaptchaException("Could not get recaptcha", redir_url);
                SolvedCaptcha solved = null;
                try {
                    solved = CaptchaSolver.solve(redir_url, sitekey, data_s);
                } catch (JsonParserException | InterruptedException e) {
                    e.printStackTrace();
                }
                formBuilder.add("g-recaptcha-response", solved.getRecaptchaResponse());
                var formReqBuilder = new okhttp3.Request.Builder().url("https://www.google.com/sorry/index").header("User-Agent", Constants.USER_AGENT).post(formBuilder.build());
                var formResponse = Constants.h2_no_redir_client.newCall(formReqBuilder.build()).execute();
                saved_cookie = HttpCookie.parse(URLUtils.silentDecode(StringUtils.substringAfter(formResponse.headers().get("Location"), "google_abuse="))).get(0);
                cookie_received = System.currentTimeMillis();
            }
            if (// call again as captcha has been solved or cookie has not expired.
            saved_cookie != null)
                execute(request);
        }
    }
    var response = new Response(resp.code(), resp.message(), resp.headers().toMultimap(), resp.body().string(), String.valueOf(resp.request().url()));
    resp.close();
    return response;
}
Also used : Caffeine(com.github.benmanes.caffeine.cache.Caffeine) ReCaptchaException(org.schabi.newpipe.extractor.exceptions.ReCaptchaException) LoadingCache(com.github.benmanes.caffeine.cache.LoadingCache) Constants(me.kavin.piped.consts.Constants) Downloader(org.schabi.newpipe.extractor.downloader.Downloader) IOException(java.io.IOException) SolvedCaptcha(me.kavin.piped.utils.obj.SolvedCaptcha) Scheduler(com.github.benmanes.caffeine.cache.Scheduler) StringUtils(org.apache.commons.lang3.StringUtils) RequestBody(okhttp3.RequestBody) TimeUnit(java.util.concurrent.TimeUnit) HttpCookie(java.net.HttpCookie) FormBody(okhttp3.FormBody) JsonParserException(com.grack.nanojson.JsonParserException) Element(org.jsoup.nodes.Element) Request(org.schabi.newpipe.extractor.downloader.Request) Jsoup(org.jsoup.Jsoup) NotNull(org.jetbrains.annotations.NotNull) Response(org.schabi.newpipe.extractor.downloader.Response) SolvedCaptcha(me.kavin.piped.utils.obj.SolvedCaptcha) Element(org.jsoup.nodes.Element) JsonParserException(com.grack.nanojson.JsonParserException) Request(org.schabi.newpipe.extractor.downloader.Request) ReCaptchaException(org.schabi.newpipe.extractor.exceptions.ReCaptchaException) Response(org.schabi.newpipe.extractor.downloader.Response) RequestBody(okhttp3.RequestBody)

Aggregations

JsonParserException (com.grack.nanojson.JsonParserException)30 JsonObject (com.grack.nanojson.JsonObject)20 ParsingException (org.schabi.newpipe.extractor.exceptions.ParsingException)18 JsonArray (com.grack.nanojson.JsonArray)11 Downloader (org.schabi.newpipe.extractor.downloader.Downloader)11 IOException (java.io.IOException)10 ReCaptchaException (org.schabi.newpipe.extractor.exceptions.ReCaptchaException)9 Response (org.schabi.newpipe.extractor.downloader.Response)8 ExtractionException (org.schabi.newpipe.extractor.exceptions.ExtractionException)8 ArrayList (java.util.ArrayList)5 MalformedURLException (java.net.MalformedURLException)4 Nonnull (javax.annotation.Nonnull)4 Page (org.schabi.newpipe.extractor.Page)3 SharedPreferences (android.content.SharedPreferences)2 URL (java.net.URL)2 DateTimeParseException (java.time.format.DateTimeParseException)2 CommentsInfoItemsCollector (org.schabi.newpipe.extractor.comments.CommentsInfoItemsCollector)2 PeertubeInstance (org.schabi.newpipe.extractor.services.peertube.PeertubeInstance)2 RegexException (org.schabi.newpipe.extractor.utils.Parser.RegexException)2 Caffeine (com.github.benmanes.caffeine.cache.Caffeine)1