Examples with RemoteInterpreterProcess org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess used on opensource projects

Search in sources :

Example 6 with RemoteInterpreterProcess

use of org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess in project SSM by Intel-bigdata.

the class ResourcePoolUtils method removeResourcesBelongsToParagraph.

public static void removeResourcesBelongsToParagraph(String noteId, String paragraphId) {
    for (InterpreterGroup intpGroup : InterpreterGroup.getAll()) {
        ResourceSet resourceSet = new ResourceSet();
        RemoteInterpreterProcess remoteInterpreterProcess = intpGroup.getRemoteInterpreterProcess();
        if (remoteInterpreterProcess == null) {
            ResourcePool localPool = intpGroup.getResourcePool();
            if (localPool != null) {
                resourceSet.addAll(localPool.getAll());
            }
            if (noteId != null) {
                resourceSet = resourceSet.filterByNoteId(noteId);
            }
            if (paragraphId != null) {
                resourceSet = resourceSet.filterByParagraphId(paragraphId);
            }
            for (Resource r : resourceSet) {
                localPool.remove(r.getResourceId().getNoteId(), r.getResourceId().getParagraphId(), r.getResourceId().getName());
            }
        } else if (remoteInterpreterProcess.isRunning()) {
            RemoteInterpreterService.Client client = null;
            boolean broken = false;
            try {
                client = remoteInterpreterProcess.getClient();
                List<String> resourceList = client.resourcePoolGetAll();
                Gson gson = new Gson();
                for (String res : resourceList) {
                    resourceSet.add(gson.fromJson(res, Resource.class));
                }
                if (noteId != null) {
                    resourceSet = resourceSet.filterByNoteId(noteId);
                }
                if (paragraphId != null) {
                    resourceSet = resourceSet.filterByParagraphId(paragraphId);
                }
                for (Resource r : resourceSet) {
                    client.resourceRemove(r.getResourceId().getNoteId(), r.getResourceId().getParagraphId(), r.getResourceId().getName());
                }
            } catch (Exception e) {
                logger.error(e.getMessage(), e);
                broken = true;
            } finally {
                if (client != null) {
                    intpGroup.getRemoteInterpreterProcess().releaseClient(client, broken);
                }
            }
        }
    }
}
Also used : InterpreterGroup(org.apache.zeppelin.interpreter.InterpreterGroup) Gson(com.google.gson.Gson) RemoteInterpreterProcess(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess) List(java.util.List)

Example 7 with RemoteInterpreterProcess

use of org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess in project zeppelin by apache.

the class ResourcePoolUtils method getAllResourcesExcept.

public static ResourceSet getAllResourcesExcept(String interpreterGroupExcludsion) {
    ResourceSet resourceSet = new ResourceSet();
    for (InterpreterGroup intpGroup : InterpreterGroup.getAll()) {
        if (interpreterGroupExcludsion != null && intpGroup.getId().equals(interpreterGroupExcludsion)) {
            continue;
        }
        RemoteInterpreterProcess remoteInterpreterProcess = intpGroup.getRemoteInterpreterProcess();
        if (remoteInterpreterProcess == null) {
            ResourcePool localPool = intpGroup.getResourcePool();
            if (localPool != null) {
                resourceSet.addAll(localPool.getAll());
            }
        } else if (remoteInterpreterProcess.isRunning()) {
            RemoteInterpreterService.Client client = null;
            boolean broken = false;
            try {
                client = remoteInterpreterProcess.getClient();
                if (client == null) {
                    // remote interpreter may not started yet or terminated.
                    continue;
                }
                List<String> resourceList = client.resourcePoolGetAll();
                Gson gson = new Gson();
                for (String res : resourceList) {
                    resourceSet.add(gson.fromJson(res, Resource.class));
                }
            } catch (Exception e) {
                logger.error(e.getMessage(), e);
                broken = true;
            } finally {
                if (client != null) {
                    intpGroup.getRemoteInterpreterProcess().releaseClient(client, broken);
                }
            }
        }
    }
    return resourceSet;
}
Also used : InterpreterGroup(org.apache.zeppelin.interpreter.InterpreterGroup) Gson(com.google.gson.Gson) RemoteInterpreterProcess(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess) List(java.util.List)

Example 8 with RemoteInterpreterProcess

use of org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess in project zeppelin by apache.

the class ResourcePoolUtils method removeResourcesBelongsToParagraph.

public static void removeResourcesBelongsToParagraph(String noteId, String paragraphId) {
    for (InterpreterGroup intpGroup : InterpreterGroup.getAll()) {
        ResourceSet resourceSet = new ResourceSet();
        RemoteInterpreterProcess remoteInterpreterProcess = intpGroup.getRemoteInterpreterProcess();
        if (remoteInterpreterProcess == null) {
            ResourcePool localPool = intpGroup.getResourcePool();
            if (localPool != null) {
                resourceSet.addAll(localPool.getAll());
            }
            if (noteId != null) {
                resourceSet = resourceSet.filterByNoteId(noteId);
            }
            if (paragraphId != null) {
                resourceSet = resourceSet.filterByParagraphId(paragraphId);
            }
            for (Resource r : resourceSet) {
                localPool.remove(r.getResourceId().getNoteId(), r.getResourceId().getParagraphId(), r.getResourceId().getName());
            }
        } else if (remoteInterpreterProcess.isRunning()) {
            RemoteInterpreterService.Client client = null;
            boolean broken = false;
            try {
                client = remoteInterpreterProcess.getClient();
                List<String> resourceList = client.resourcePoolGetAll();
                Gson gson = new Gson();
                for (String res : resourceList) {
                    resourceSet.add(gson.fromJson(res, Resource.class));
                }
                if (noteId != null) {
                    resourceSet = resourceSet.filterByNoteId(noteId);
                }
                if (paragraphId != null) {
                    resourceSet = resourceSet.filterByParagraphId(paragraphId);
                }
                for (Resource r : resourceSet) {
                    client.resourceRemove(r.getResourceId().getNoteId(), r.getResourceId().getParagraphId(), r.getResourceId().getName());
                }
            } catch (Exception e) {
                logger.error(e.getMessage(), e);
                broken = true;
            } finally {
                if (client != null) {
                    intpGroup.getRemoteInterpreterProcess().releaseClient(client, broken);
                }
            }
        }
    }
}
Also used : InterpreterGroup(org.apache.zeppelin.interpreter.InterpreterGroup) Gson(com.google.gson.Gson) RemoteInterpreterProcess(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess) List(java.util.List)

Example 9 with RemoteInterpreterProcess

use of org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess in project zeppelin by apache.

the class InterpreterSettingManager method removeResourcesBelongsToParagraph.

public void removeResourcesBelongsToParagraph(String noteId, String paragraphId) {
    for (ManagedInterpreterGroup intpGroup : getAllInterpreterGroup()) {
        ResourceSet resourceSet = new ResourceSet();
        RemoteInterpreterProcess remoteInterpreterProcess = intpGroup.getRemoteInterpreterProcess();
        if (remoteInterpreterProcess == null) {
            ResourcePool localPool = intpGroup.getResourcePool();
            if (localPool != null) {
                resourceSet.addAll(localPool.getAll());
            }
            if (noteId != null) {
                resourceSet = resourceSet.filterByNoteId(noteId);
            }
            if (paragraphId != null) {
                resourceSet = resourceSet.filterByParagraphId(paragraphId);
            }
            for (Resource r : resourceSet) {
                localPool.remove(r.getResourceId().getNoteId(), r.getResourceId().getParagraphId(), r.getResourceId().getName());
            }
        } else if (remoteInterpreterProcess.isRunning()) {
            try {
                List<String> resourceList = remoteInterpreterProcess.callRemoteFunction(client -> client.resourcePoolGetAll());
                for (String res : resourceList) {
                    resourceSet.add(Resource.fromJson(res));
                }
                if (noteId != null) {
                    resourceSet = resourceSet.filterByNoteId(noteId);
                }
                if (paragraphId != null) {
                    resourceSet = resourceSet.filterByParagraphId(paragraphId);
                }
                for (final Resource r : resourceSet) {
                    remoteInterpreterProcess.callRemoteFunction(client -> {
                        client.resourceRemove(r.getResourceId().getNoteId(), r.getResourceId().getParagraphId(), r.getResourceId().getName());
                        return null;
                    });
                }
            } catch (Exception e) {
                LOGGER.error(e.getMessage());
            }
        }
    }
}
Also used : ZeppelinConfiguration(org.apache.zeppelin.conf.ZeppelinConfiguration) Arrays(java.util.Arrays) TypeToken(com.google.gson.reflect.TypeToken) RemoteInterpreterProcess(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess) Resource(org.apache.zeppelin.resource.Resource) Proxy(org.eclipse.aether.repository.Proxy) URL(java.net.URL) LoggerFactory(org.slf4j.LoggerFactory) ResourcePool(org.apache.zeppelin.resource.ResourcePool) AuthenticationInfo(org.apache.zeppelin.user.AuthenticationInfo) StringUtils(org.apache.commons.lang3.StringUtils) GsonBuilder(com.google.gson.GsonBuilder) Metrics(io.micrometer.core.instrument.Metrics) ManagedAttribute(org.eclipse.jetty.util.annotation.ManagedAttribute) DirectoryStream(java.nio.file.DirectoryStream) URLClassLoader(java.net.URLClassLoader) Matcher(java.util.regex.Matcher) CLUSTER_INTP_SETTING_EVENT_TOPIC(org.apache.zeppelin.cluster.ClusterManagerServer.CLUSTER_INTP_SETTING_EVENT_TOPIC) Gson(com.google.gson.Gson) ConfVars(org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars) Job(org.apache.zeppelin.scheduler.Job) Map(java.util.Map) ClusterEvent(org.apache.zeppelin.cluster.event.ClusterEvent) Dependency(org.apache.zeppelin.dep.Dependency) ManagedObject(org.eclipse.jetty.util.annotation.ManagedObject) Path(java.nio.file.Path) Tags(io.micrometer.core.instrument.Tags) ReflectionUtils(org.apache.zeppelin.util.ReflectionUtils) Gauge(io.micrometer.core.instrument.Gauge) ImmutableMap(com.google.common.collect.ImmutableMap) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) ConfigStorage(org.apache.zeppelin.storage.ConfigStorage) Collectors(java.util.stream.Collectors) ClusterManagerServer(org.apache.zeppelin.cluster.ClusterManagerServer) RemoteAngularObjectRegistry(org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry) DependencyResolver(org.apache.zeppelin.dep.DependencyResolver) List(java.util.List) Type(java.lang.reflect.Type) Pattern(java.util.regex.Pattern) AngularObjectRegistry(org.apache.zeppelin.display.AngularObjectRegistry) RegisteredInterpreter(org.apache.zeppelin.interpreter.Interpreter.RegisteredInterpreter) RemoteInterpreterProcessListener(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcessListener) ClusterEventListener(org.apache.zeppelin.cluster.event.ClusterEventListener) ArrayUtils(org.apache.commons.lang3.ArrayUtils) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) Inject(javax.inject.Inject) ClusterMessage(org.apache.zeppelin.cluster.event.ClusterMessage) HashSet(java.util.HashSet) AngularObjectRegistryListener(org.apache.zeppelin.display.AngularObjectRegistryListener) LinkedList(java.util.LinkedList) NoteEventListener(org.apache.zeppelin.notebook.NoteEventListener) Paragraph(org.apache.zeppelin.notebook.Paragraph) ResourceSet(org.apache.zeppelin.resource.ResourceSet) Logger(org.slf4j.Logger) ApplicationEventListener(org.apache.zeppelin.helium.ApplicationEventListener) MalformedURLException(java.net.MalformedURLException) Files(java.nio.file.Files) ApplicationState(org.apache.zeppelin.notebook.ApplicationState) Note(org.apache.zeppelin.notebook.Note) Meter(io.micrometer.core.instrument.Meter) FileUtils(org.apache.commons.io.FileUtils) IOException(java.io.IOException) RecoveryStorage(org.apache.zeppelin.interpreter.recovery.RecoveryStorage) FileInputStream(java.io.FileInputStream) Notebook(org.apache.zeppelin.notebook.Notebook) InputStreamReader(java.io.InputStreamReader) File(java.io.File) ParagraphTextParser(org.apache.zeppelin.notebook.ParagraphTextParser) RemoteRepository(org.eclipse.aether.repository.RemoteRepository) Authentication(org.eclipse.aether.repository.Authentication) Paths(java.nio.file.Paths) Preconditions(com.google.common.base.Preconditions) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Comparator(java.util.Comparator) Collections(java.util.Collections) InputStream(java.io.InputStream) Resource(org.apache.zeppelin.resource.Resource) RemoteInterpreterProcess(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess) ResourcePool(org.apache.zeppelin.resource.ResourcePool) List(java.util.List) ArrayList(java.util.ArrayList) LinkedList(java.util.LinkedList) ResourceSet(org.apache.zeppelin.resource.ResourceSet) MalformedURLException(java.net.MalformedURLException) IOException(java.io.IOException)

Example 10 with RemoteInterpreterProcess

use of org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess in project SSM by Intel-bigdata.

the class ResourcePoolUtils method getAllResourcesExcept.

public static ResourceSet getAllResourcesExcept(String interpreterGroupExcludsion) {
    ResourceSet resourceSet = new ResourceSet();
    for (InterpreterGroup intpGroup : InterpreterGroup.getAll()) {
        if (interpreterGroupExcludsion != null && intpGroup.getId().equals(interpreterGroupExcludsion)) {
            continue;
        }
        RemoteInterpreterProcess remoteInterpreterProcess = intpGroup.getRemoteInterpreterProcess();
        if (remoteInterpreterProcess == null) {
            ResourcePool localPool = intpGroup.getResourcePool();
            if (localPool != null) {
                resourceSet.addAll(localPool.getAll());
            }
        } else if (remoteInterpreterProcess.isRunning()) {
            RemoteInterpreterService.Client client = null;
            boolean broken = false;
            try {
                client = remoteInterpreterProcess.getClient();
                List<String> resourceList = client.resourcePoolGetAll();
                Gson gson = new Gson();
                for (String res : resourceList) {
                    resourceSet.add(gson.fromJson(res, Resource.class));
                }
            } catch (Exception e) {
                logger.error(e.getMessage(), e);
                broken = true;
            } finally {
                if (client != null) {
                    intpGroup.getRemoteInterpreterProcess().releaseClient(client, broken);
                }
            }
        }
    }
    return resourceSet;
}
Also used : InterpreterGroup(org.apache.zeppelin.interpreter.InterpreterGroup) Gson(com.google.gson.Gson) RemoteInterpreterProcess(org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess) List(java.util.List)

Aggregations

RemoteInterpreterProcess (org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess)10 Gson (com.google.gson.Gson)6 List (java.util.List)6 InterpreterGroup (org.apache.zeppelin.interpreter.InterpreterGroup)4 IOException (java.io.IOException)3 TypeToken (com.google.gson.reflect.TypeToken)2 File (java.io.File)2 ByteBuffer (java.nio.ByteBuffer)2 ArrayList (java.util.ArrayList)2 Collections (java.util.Collections)2 LinkedList (java.util.LinkedList)2 Map (java.util.Map)2 TException (org.apache.thrift.TException)2 TTransportException (org.apache.thrift.transport.TTransportException)2 InterpreterRPCException (org.apache.zeppelin.interpreter.thrift.InterpreterRPCException)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Preconditions (com.google.common.base.Preconditions)1 ImmutableMap (com.google.common.collect.ImmutableMap)1 GsonBuilder (com.google.gson.GsonBuilder)1 Gauge (io.micrometer.core.instrument.Gauge)1
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial