Ram Memories
Main Categories
Find Products
Featured keywords
,,,,,,,,,,
Check these
Loading...
Browsing the internet
for great deals on Ram Memories?
Then you 've come to the best place! Just choose one of our selected categories to find a list of the most value for money products that you can find on line.
A wide variety of Ram Memories is waiting for you only here at cheapest online prices. Different types are available for you from top name brands.
Price: $269.99
Now for only: $52.99
Great looking, great overclocking memory at a great price Like the legendary Dominator, enthusiast-grade Vengeance DRAM is designed for stability, stringently factory-tested, and backed by our limited...
Price: $100.99
Now for only: $58.60
Corsair Vengeance 8GB (2x4GB) DDR3 1600MHz PC3 SODIMM Laptop Memory 204 Pin CMSX8GX3M2A1600C9
Price: $79.99
Now for only: $47.00
Corsair 8GB (2x 4GB) 1333mhz PC3--pin DDR3 SODIMM Laptop Memory Kit CMSO8GX3M2A1333C9
Corsair Memory Kits: compatibility and support you can count on If you’re building or upgrading a des...
Price: $259.99
Now for only: $51.99
16GB Kit (4x4GB) PC2-5300 DDR2 ECC Fully Buffered Server MemorySpecs:DIMMDDR2PC2-Mhz)240 Pin1.8VECC (Fully Buffered)Designed for DELL Servers and Workstations:
Dell PowerEdge 50 ...
Price: $159.99
Now for only: $88.95
BLS2KIT8G3D160ADS1S00 is a 16GB kit consisting of (2) 8GB DDR3 modules that operates at speeds up to 1600 MT/s and has a CL9 latency with extended timings of 9-9-9-24. It is Performance and is non-ECC...
Price: $84.99
Now for only: $49.75
CT2CP5B is an 8GB kit consisting of (2) 4GB DDR3 Notebook modules that operates at speeds up to 1600 MT/s and has a CL11 latency. It is dual voltage and can operate at 1.35V or 1.5V. It is Un...
Price: $164.99
Now for only: $89.99
CT2KIT0B is a 16GB kit consisting of (2) 8GB DDR3 1.35V Notebook module that operates at speeds up to 1600 MT/s and has a CL11 latency. It is dual voltage and can operate at 1.35V and 1.5V. ...
Price: $179.99
Now for only: $95.49
Kingston Technology HyperX Impact Kit 1600MHz DDR3L CL9 SODIMM 1.35V Laptop Memory HX316LS9IBK2/16 Black
For higher performance upgrades, increased in-game FPS and an overall faster system.View large...
Price: $257.00
Now for only: $85.99
For higher performance upgrades, increased in-game FPS and an overall faster system.
Effortless Upgrade, Relentless Style Get in the game with HyperX FURY. Even newbies get up to speed fast, since FU...
Price: $359.99
Now for only: $106.99
32GB Kit (8x4GB) PC2-5300 DDR2 ECC Fully Buffered Server MemorySpecs:DIMMDDR2PC2-Mhz)240 Pin1.8VECC (Fully Buffered)Designed for DELL Servers and Workstations:
Dell PowerEdge 50 ...
Price: $165.99
Now for only: $89.99
CT2K8G3S160BM is a 16GB kit consisting of (2) 8GB DDR3L (DDR3 low voltage) 1.35V Notebook module that operates at speeds up to 1600 MT/s and has a CL11 latency. It is dual voltage and can operate at 1...
Price: $89.99
Now for only: $53.99
CT2C4G3S1067M is an 8 GB kit consisting of (2) 4 GB DDR3 Notebook modules that operates at speeds up to 1066 MT/s and has a CL7 latency. It is Unbuffered and is non-ECC. It conforms to the industry st...
Price: $100.99
Now for only: $49.90
Crucial Ballistix Sport is a 8GB kit consisting of (2) 4GB DDR3 modules that operates at speeds up to 1600 MT/s and has a CL9 latency with extended timings of 9-9-9-24. It is Performance and is non-EC...
Price: $189.99
Now for only: $103.24
Corsair Mac Memory Upgrade Kits: Guaranteed Performance, Compatibility, and Support Corsair's 4GB, 8GB, and 16GB Mac memory upgrade kits will help your system work better. With hassle-free compatibili...
Price: $177.31
Now for only: $104.99
Great looking, great overclocking memory at a great price Like the legendary Dominator, enthusiast-grade Vengeance DRAM is designed for stability, stringently factory-tested, and backed by our limited...
Price: $99.99
Now for only: $58.99
Corsair Mac Memory Upgrade Kits: Guaranteed Performance, Compatibility, and Support Corsair's 4GB and 8GB Mac memory upgrade kits will help your system work better. With hassle-free compatibility and ...
Price: $99.99
Now for only: $51.99
Kingston's KTA-MB667K2/4GR is a 4GB Kit (2pcs of 256M X 64) Non-ECC 667MHz 200-pin Unbuffered SODIMM (SDRAM-DDR2, 1.8V, CL5, FBGA, Gold). This memory is designed for the following systems: iMac Intel ...
Price: $269.99
Now for only: $52.99
Great looking, great overclocking memory at a great price Like the legendary Dominator, enthusiast-grade Vengeance DRAM is designed for stability, stringently factory-tested, and backed by our limited...
Price: $179.99
Now for only: $103.24
CORSAIR Mac Memory, 16GB 1600MHz DDR3 memory kit for Apple iMac, MacBook, MacBook Pro, IMac and Mac mini. Tested at Apples compatibility lab to ensure functionality with all mid 2011 and new systems. ...
Price: $100.99
Now for only: $57.99
Corsair Vengeance 8GB (2x4GB) DDR3 1600MHz PC3 SODIMM Laptop Memory 204 Pin CMSX8GX3M2A1600C9
« Previous1...
Useful Info
Online services have conquered our daily life and nowadays the majority of sevices can be offered online via web. Therefor, it is not surprising that purchasing things via net has recently become very popular. When shopping via web we take advantage of many profits. We have the opportunity to make our shopping any day, anytime while we may effortlessly compare items of different shops and vendors. We can complete an online search to find more information while frequently we get to know evaluations by prior consumers.
But regrettably, besides all the aforementioned advantages there are several perils when shopping online. Don't panic! If you comply with some easy steps you won't have any reason to anguish about. So here we are.
Firstly, you need to be certain that your personal computer, laptop, tablet, mobile or any other device, through which you will do the purchase, is secured. That means that it is up to date with all the required antivirus filters and security as well as security program and updated patches and net browser security with the trusted websites and protection level at high.
At no time shop goods from an untrusted site and never neglect to check the annotations of former consumers. Furthermore, you must be acquainted with all the steps that will take place in order to do the buy. In other words you must know the transport charges, prices and policies and the return or replacement terms.
This page has partnered , so clicking on the items will lead you to the Amazon page.
As soon as you finish the reciprocation, never forget to take a duplicate of the purchase log and facts such as cost, confirmation receipt, policies and condition of the sale etc. Amazon offers you the choice to track and trace your buys and to be aware of which steps have been done.
Finally, be certain that you will never disclose your private information, such as SSN, codes or any other account data, when completing any inquiry or in reply to any e-mail.
Other Products
Crucial 4GB DDR3/DDR3L 1066 MT/s PC3--Pin Memory for Mac CT4G3S1067M,
Crucial 2GB DDR3/DDR3L 1066 MT/s (PC3-8500) CL7 SODIMM 204-Pin for Mac CT2G3S1067M,
Crucial 8GB Kit (4GB x 2) DDR3/DDR3L-1066 MT/s (PC3-8500) CL7 204-Pin SODIMM Memory Upgrade for MAC CT2K4G3S1067M / CT2C4G3S1067M,
Crucial 4GB kit (2GB x 2) DDR3/DDR3L-1066 MT/s (PC3-8500) CL7 204-Pin SODIMM memory upgrade for Mac CT2K2G3S1067M / CT2C2G3S1067M,
Crucial 8GB Kit (4GBx2) DDR3 1066 MT/s (PC3-8500) CL7 SODIMM 204-Pin Notebook Memory Modules CT2CP5,
Kingston Apple 2GB Kit (2x1GB Modules) 667MHz DDR2 SoDimm iMac and Macbook Memory (KTA-MB667K2/2GR),
Kingston Apple 4GB Kit (2x2GB Modules) 667MHz DDR2 SoDimm iMac and Macbook Memory (KTA-MB667K2/4GR),
Crucial 4GB Single DDR3 1333 MT/s (PC3-10600) CL9 204-Pin 1.35V/1.5V SODIMM Memory For Mac CT4G3S1339M,
Crucial 4GB Kit (2GBx2) DDR3-1333 MT/s (PC3-10600) CL9 204-Pin SODIMM Memory For Mac CT2K2G3S1339M / CT2C2G3S1339M,
Crucial 8GB Kit (4GBx2) DDR3 1333 MT/s (PC3-10600) CL9 204-Pin SODIMM Memory For Mac CT2K4G3S1339M (Notebook),11118人阅读
转载请注明作者，谢谢支持！
一、环境准备
测试环境使用的cdh提供的quickstart
hadoop版本：2.5.0-cdh5.2.0
spark版本：1.1.0
二、Hello Spark
将/usr/lib/spark/examples/lib/spark-examples-1.1.0-cdh5.2.0-hadoop2.5.0-cdh5.2.0.jar
移动到/usr/lib/spark/lib/spark-examples-1.1.0-cdh5.2.0-hadoop2.5.0-cdh5.2.0.jar
./bin/run-example SparkPi 10
日志分析：
程序检查ip,host,SecurityManager
启动sparkDriver。通过akka工具启动一个tcp监听&
[akka.tcp://sparkDriver@192.168.128.131:42960]
启动MapOutputTracker，BlockManagerMaster
启动一个block manager，也就是ConnectionManagerId(192.168.128.131,41898)，其中包含一个MemoryStore
通过netty启动一个HTTP
file server：SocketConnector@0.0.0.0:55161
启动一个sparkUI：http://192.168.128.131:4040
通过http上传本地程序运行Jar包
HeartbeatReceiver: akka.tcp://sparkDriver@192.168.128.131:42960/user/HeartbeatReceiver
Starting job: reduce
分析中job，有stage
0 (MappedRDD[1])
添加并启动运行task&
Submitting 10 missing tasks from Stage 0
通过http协议获取程序jar包，并添加到classloader
完成task后，将结果发送到driver
scheduler.DAGScheduler完成Stage的所有task
在localhost的scheduler.TaskSetManager收集完成的task
job finished
Stop Spark Web UI
Stop DAGScheduler
MapOutputTrackerActor stopped
ConnectionManager
MemoryStore cleared
BlockManager stopped
Shutting down remote daemon.
Successfully stopped SparkContext
三、cluster mode
运行流程：：
SparkContext&连接cluster
Manager (either Spark’s own standalone cluster manager or Mesos/YARN),
spark Application向Cluster
Manager请求资源 executors (运行计算和存储数据的线程)
将程序Jar包或者python程序分发到executors
SparkContext发送tasks到executors上运行
Cluster Manager
Spark 内置的cluster manager，可以快速启动一个集群
&一个通用的Cluster
manger，可以运行hadoop的Mapreduce和其他Service
applications &Hadoop
2中的Clustger Manager
Application
User program built on Spark. Consists of a&driver program&and&executors&on
the cluster.
Application jar
A jar containing the user's Spark application. In some cases users will want to create an &uber jar& containing their application along with its dependencies. The user's jar should never include Hadoop or Spark libraries, however, these will
be added at runtime.
Driver program
The process running the main() function of the application and creating the SparkContext
Cluster manager
An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)
Deploy mode
Distinguishes where the driver process runs. In &cluster& mode, the framework launches the driver inside of the cluster. In &client& mode, the submitter launches the driver outside of the cluster.
Worker node
Any node that can run application code in the cluster
A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.
A unit of work that will be sent to one executor
A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g.&save,&collect);
you'll see this term used in the driver's logs.
Each job gets divided into smaller sets of tasks called&stages&that depend on each other (similar to the map
and reduce stages in MapReduce); you'll see this term used in the driver's logs.
mode 运行模式
yarn集群方式运行
spark-submit
--classcom.wankun.sparktest.WordCount
--masteryarn-cluster
target/sparktest-1.0.0.jar/tmp/test1 2
运行命令：
yarn-cluster
spark-submit --classcom.wankun.sparktest.WordCount --master yarn-cluster --driver-memory 385m--executor-memory 410m&target/sparktest-1.0.0.jar /tmp/test1 2
运行成功后，无任何输出，输出都在日志中
程序的运行有yarn来控制，spark只是检测程序的状态，状态为success，即运行成功
yarn-client
spark-submit--class com.wankun.sparktest.WordCount --master yarn-client --driver-memory 385m --executor-memory 410m& target/sparktest-1.0.0.jar
/tmp/test1 2
yarn-cluster
的driver programcontainer
是在集群里的，yarn-client
的driver programcontainer
是spark在集群外自己启动的
运行原理：
scheduler.DAGScheduler，scheduler.TaskSetManager，cluster.YarnClusterScheduler
spark向RM申请一个Container作为调度container（此时启动的SparkUI端口随机）
请求Executors（默认2个，
Container request (host: Any, priority: 1, capability: &memory:1408, vCores:1&）
Received new token for : quickstart.cloudera:48622
根据resources and environment and commands, open proxy
start progress reporter
在YarnClusterSchedulerBackend,BlockManagerMasterActor,MemoryStore等服务启动
在SparkContext中Starting
TasksetManager中starting
task with TID 0,1
任务调度由scheduler.DAGScheduler执行，根据job和job中tasks进行任务执行，Taskset
will be removed ,when completed
job全部执行结束，Stopped
Spark web UI,Stopping DAGScheduler,Shutting down all
executors应该是可以重用
executors通过CoarseGrainedExecutorBackend
获取分配的任务，关闭的时候，Driver commanded a shutdown
在关闭http file server进程时，遇到错误
14/11/05 20:17:40 WARN thread.QueuedThreadPool: 1 threads could not be stopped
14/11/05 20:17:40 INFO thread.QueuedThreadPool: Couldn't stop Thread[qtp Acceptor0 SocketConnector@0.0.0.0:39213,5,main]
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at java.net.PlainSocketImpl.socketAccept(Native Method)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at java.net.ServerSocket.implAccept(ServerSocket.java:530)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at java.net.ServerSocket.accept(ServerSocket.java:498)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
14/11/05 20:17:41 INFO thread.QueuedThreadPool:& at java.lang.Thread.run(Thread.java:745)
14/11/05 20:17:41 INFO network.ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2cc51248
14/11/05 20:17:41 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/11/05 20:17:41 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(quickstart.cloudera,48234)
14/11/05 20:17:41 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(quickstart.cloudera,52620)
14/11/05 20:17:42 INFO network.ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2cc51248
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/11/05 20:17:42 INFO network.ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@69aaccdf
14/11/05 20:17:42 INFO network.ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@69aaccdf
java.nio.channels.CancelledKeyException
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
方式一：yarnlogs -applicationId application_5_0002
方式二：通过Rm:8088端口进入Spark
history Server:18088端口查看
配置spark-defaults.conf
中jobhistory中的配置
spark.eventLog.enabled=true
spark.eventLog.dir=hdfs:///user/spark/applicationHistory
spark.yarn.historyServer.address=http://quickstart.cloudera:18088
启动 spark-history-server
服务 此时，在yarn
集群中提交的服务日志会上传的hdfs上，在RM:8088页面中可以直接调整到spark页面进行查看
提交参数：
spark-submit --class com.wankun.sparktest.WordCount --master yarn-cluster --driver-memory 385m --executor-memory 410m& target/sparktest-1.0.0.jar /tmp/test1
实际上，driverExecutor和task Executor 占用那个的内存显示并没有这么多，不清楚什么原因
14/11/05 11:56:14ERROR yarn.Client: Error: Executor memory sizemust be greater than: 384
Exception in thread&main& java.lang.IllegalArgumentException: Usage:org.apache.spark.deploy.yarn.Client [options]
& --jar JAR_PATH&&&&&&&&&&&& Path to your application's JARfile (required in yarn-cluster mode)
& --class CLASS_NAME&&&&&&&& Name of your application's main class(required)
& --arg ARGS&&&&&&&&&&&&&&&& Argument to be passed to yourapplication's main class.
&&&&&&&&&&&&&&&&&&&&&&&&&&&& Multipleinvocations are possible, each will be passed in order.
& --num-executors NUM&&&&&&& Number of executors to start (Default:2)
& --executor-cores NUM&&&&&& Number of cores for the executors(Default: 1).
& --driver-memory MEM&&&&&&& Memory for driver (e.g. 1000M, 2G)(Default: 512 Mb)
& --executor-memory MEM&&&&& Memory per executor (e.g. 1000M, 2G)(Default: 1G)
& --name NAME&&&&&&&&&&&&&&& The name of your application(Default: Spark)
& --queue QUEUE&&&&&&&&&&&&& The hadoop queue to use forallocation requests (Default: 'default')
& --addJars jars&&&&&&&&&&&& Comma separated list of local jarsthat want SparkContext.addJar to work with.
& --files files&&&&&&&&&&&&& Comma separated list of files tobe distributed with the job.
& --archives archives&&&&&&& Comma separated list of archives to bedistributed with the job.
将spark的hadoop类库上传到hdfs上，省的每次都上传
hdfs dfs -mkdir -p /user/spark/share/lib
hadoop dfs -put /usr/lib/spark/assembly/lib/spark-assembly-1.1.0-cdh5.2.0-hadoop2.5.0-cdh5.2.0.jar /user/spark/share/lib/spark-assembly.jar
&hadoop dfs -chmod -R 777 /user/spark/
在spark-env.sh中配置
export SPARK_JAR=hdfs://quickstart.cloudera:8020/user/spark/share/lib/spark-assembly.jar
五、spark cluster
mode 运行模式
启动服务：spark-history-server& spark-master&&&&&&&&& spark-worker
spark-master
监控页面：
&application
detail 页面，如果有多个sparkContext，端口依次递增（如…)，程序结束后，关闭。页面主要内容
stages and tasks,
&RDD sizes and memory usage
Environmental
192.168.128.131 7077 master通信端口
spark-worker
监控页面：
192.168.128.131 7078 worker通信端口
spark-history-server
监控页面：
spark master和worker之间的通信使用的是akka，tcp协议。例如：[akka.tcp://sparkWorker@192.168.128.131:7078]
备注：测试时，因为master绑定在了192.168.128.131
ip上了，所以必须在/etc/spark/con/spark-env.sh配置文件配置上exportSPARK_MASTER_IP=192.168.128.131
spark-env.sh主要配置
export STANDALONE_SPARK_MASTER_HOST=&192.168.128.131&
export SPARK_MASTER_IP=$STANDALONE_SPARK_MASTER_HOST
### Let's run everything with JVM runtime, instead of Scala
export SPARK_LAUNCH_WITH_SCALA=0
export SPARK_LIBRARY_PATH=${SPARK_HOME}/lib
export SCALA_LIBRARY_PATH=${SPARK_HOME}/lib
export SPARK_MASTER_WEBUI_PORT=18080
export SPARK_MASTER_IP=&192.168.128.131&
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=100m
export SPARK_WORKER_PORT=7078
export SPARK_WORKER_INSTANCES=1
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_DIR=/var/run/spark/work
export SPARK_LOG_DIR=/var/log/spark
export SPARK_PID_DIR='/var/run/spark/'
if [ -n &$HADOOP_HOME& ]; then
& export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}
注意在cloudera提供的虚拟机中的配置文件有如下问题：
第一、master的7077端口并未绑定在0.0.0.0上，第二，HADOOP_CONF_DIR写错了，写成了etc/hadoop/conf。
第二、在/etc/hosts中将hostname配置上外网口ip,否则会造成master和worker通信失败，或者job无法正常提交的问题，提交job时也要使用hostname提交
六、spark-submit
spark-submit
--classcom.wankun.sparktest.JavaWordCount
--masterspark://quickstart.cloudera:7077
target/sparktest-1.0.0.jar/tmp/test1 2
其余常用参数：
& --executor-memory 20G
& --total-executor-cores 100
&--master yarn-cluster
\ &# can also be `yarn-client` for clientmode
\# Run application locally on 8 cores
&--master yarn-cluster
\ &# can also be `yarn-client` for clientmode
Master URLs
The master URL passed to Spark can be in one of thefollowing formats:
Master URL
Run Spark locally with one worker thread (i.e. no parallelism at all).
Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).
Run Spark locally with as many worker threads as logical cores on your machine.
spark://HOST:PORT
Connect to the given&&master. The port must be whichever one your master is configured to use, which is 7077 by default.
mesos://HOST:PORT
Connect to the given&&cluster.
The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use&mesos://zk://....
yarn-client
Connect to a&cluster
in client mode. The cluster location will be found based on the HADOOP_CONF_DIR variable.
yarn-cluster
Connect to a&cluster
in cluster mode. The cluster location will be found based on HADOOP_CONF_DIR.
Transformations
The following table listssome of the common transformations supported by Spark. Refer to the RDD API doc(,&,&)
and pair RDD functions doc (,&)
for details.
Transformation
Return a new distributed dataset formed by passing each element of the source through a function&func.
filter(func)
Return a new dataset formed by selecting those elements of the source on which&func&returns true.
flatMap(func)
Similar to map, but each input item can be mapped to 0 or more output items (so&func&should return a Seq rather
than a single item).
mapPartitions(func)
Similar to map, but runs separately on each partition (block) of the RDD, so&func&must be of type Iterator&T&
=& Iterator&U& when running on an RDD of type T.
mapPartitionsWithIndex(func)
Similar to mapPartitions, but also provides&func&with an integer value representing the index of the partition,
so&func&must be of type (Int, Iterator&T&) =& Iterator&U& when running on an RDD of type T.
sample(withReplacement,fraction,&seed)
Sample a fraction&fraction&of the data, with or without replacement, using a given random number generator
union(otherDataset)
Return a new dataset that contains the union of the elements in the source dataset and the argument.
intersection(otherDataset)
Return a new RDD that contains the intersection of elements in the source dataset and the argument.
distinct([numTasks]))
Return a new dataset that contains the distinct elements of the source dataset.
groupByKey([numTasks])
When called on a dataset of (K, V) pairs, returns a dataset of (K, Iterable&V&) pairs.&
you are grouping in order to perform an aggregation (such as a sum or average) over each key, using&reduceByKey&or&combineByKey&will
yield much better performance.&
default, the level of parallelism in the output depends on the number of partitions of the parent RDD. You can pass an optional&numTasks&argument
to set a different number of tasks.
reduceByKey(func, [numTasks])
When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using
the given reduce function&func,
which must be of type (V,V) =& V. Like ingroupByKey, the number of
reduce tasks is configurable through an optional second argument.
aggregateByKey(zeroValue)(seqOp,&combOp,
[numTasks])
When called on a dataset of (K, V) pairs, returns a dataset of (K, U) pairs where the values for each key are aggregated using
the given combine functions and a neutral &zero& value. Allows an aggregated value type that is different than the input value type, while avoiding unnecessary allocations. Like in&groupByKey,
the number of reduce tasks is configurable through an optional second argument.
sortByKey([ascending], [numTasks])
When called on a dataset of (K, V) pairs where K implements Ordered, returns a dataset of (K, V) pairs sorted by keys in ascending
or descending order, as specified in the boolean&ascending&argument.
join(otherDataset, [numTasks])
When called on datasets of type (K, V) and (K, W), returns a dataset of (K, (V, W)) pairs with all pairs of elements for each
key. Outer joins are also supported through&leftOuterJoin&and&rightOuterJoin.
cogroup(otherDataset, [numTasks])
When called on datasets of type (K, V) and (K, W), returns a dataset of (K, Iterable&V&, Iterable&W&) tuples. This operation
is also called&groupWith.
cartesian(otherDataset)
When called on datasets of types T and U, returns a dataset of (T, U) pairs (all pairs of elements).
pipe(command,&[envVars])
Pipe each partition of the RDD through a shell command, e.g. a Perl or bash script. RDD elements are written to the process's stdin and lines output to its stdout are returned as an RDD of strings.
coalesce(numPartitions)
Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset.
repartition(numPartitions)
Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network.
JavaPairRDD ---& JavaPairRDD
JavaPairRDD&Integer, Integer& tc;
JavaPairRDD&Integer, Integer& edges = tc.mapToPair(
&&&&& new PairFunction&Tuple2&Integer, Integer&, Integer, Integer&() {
&&&&&&& @Override
&&&&&&& public Tuple2&Integer, Integer& call(Tuple2&Integer, Integer& e) {
&&&&&&&&& return new Tuple2&Integer, Integer&(e._2(), e._1());
The following table listssome of the common actions supported by Spark. Refer to the RDD API doc (,&,&)
and pair RDD functions doc (,&)
for details.
reduce(func)
Aggregate the elements of the dataset using a function&func&(which takes two arguments and returns one). The
function should be commutative and associative so that it can be computed correctly in parallel.
a1,a2 --& b1
a2,a3 --& b2
a3,a4 --& b3
Return all the elements of the dataset as an array at the driver program. This is usually useful after a filter or other operation that returns a sufficiently small subset of the data.
Return the number of elements in the dataset.
Return the first element of the dataset (similar to take(1)).
Return an array with the first&n&elements of the dataset. Note that this is currently not executed in parallel.
Instead, the driver program computes all the elements.
takeSample(withReplacement,num, [seed])
Return an array with a random sample of&num&elements of the dataset, with or without replacement, optionally
pre-specifying a random number generator seed.
takeOrdered(n,&[ordering])
Return the first&n&elements of the RDD using either their natural order or a custom comparator.
saveAsTextFile(path)
Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in
saveAsSequenceFile(path)&
(Java and Scala)
Write the elements of the dataset as a Hadoop SequenceFile in a given path in the local filesystem, HDFS or any other Hadoop-supported file system. This is available on RDDs of key-value pairs that either implement Hadoop's Writable interface.
In Scala, it is also available on types that are implicitly convertible to Writable (Spark includes conversions for basic types like Int, Double, String, etc).
saveAsObjectFile(path)&
(Java and Scala)
Write the elements of the dataset in a simple format using Java serialization, which can then be loaded usingSparkContext.objectFile().
countByKey()
Only available on RDDs of type (K, V). Returns a hashmap of (K, Int) pairs with the count of each key.
foreach(func)
Run a function&func&on each element of the dataset. This is usually done for side effects such as updating
an accumulator variable (see below) or interacting with external storage systems.
结果是一个数据，例如，正数，数组，对象等
transformation结果是一个RDD，完成从一个RDD到另一个RDD的转换
常用工具类说明：
JavaRDD&D&
JavaPairRDD&K,V&
Tuple2(K,V&&
类似与map中的一个entry&e._1()&
Scala API doc
& apache spark mailing list
spark examples
版权声明：本文为博主原创文章，未经博主允许不得转载。
* 以上用户言论只代表其个人观点，不代表CSDN网站的观点或立场
访问：18797次
排名：千里之外
原创：32篇
(3)(2)(1)(2)(10)(10)(5)}