2014년 8월 13일 수요일
Hadoop Illuminated
reference link :
http://hadoopilluminated.com/
http://hadoopilluminated.com/hadoop_illuminated/hadoop-illuminated.pdf
http://hadoopilluminated.com/hadoop_illuminated/Hadoop_Use_Cases.html
2014년 1월 13일 월요일
apache hama?
http://www.slideshare.net/udanax/introduction-of-apache-hama-2011
http://wiki.apache.org/hama/
http://hama.apache.org/
Apache Hama is an Apache Top-Level open source project, allowing you to do advanced analytics beyond MapReduce.
Many data analysis techniques such as machine learning and graph algorithms require iterative computations, this is where Bulk Synchronous Parallel model can be more effective than "plain" MapReduce. Therefore To run such iterative data analysis applications more efficiently, Hama offers pure Bulk Synchronous Parallel computing engine.
Hama can make jobs that take hours in Hadoop run in minutes. For details, see the Performance Benchmarks. ??
http://wiki.apache.org/hama/
http://hama.apache.org/
Apache Hama is an Apache Top-Level open source project, allowing you to do advanced analytics beyond MapReduce.
Many data analysis techniques such as machine learning and graph algorithms require iterative computations, this is where Bulk Synchronous Parallel model can be more effective than "plain" MapReduce. Therefore To run such iterative data analysis applications more efficiently, Hama offers pure Bulk Synchronous Parallel computing engine.
Hama can make jobs that take hours in Hadoop run in minutes. For details, see the Performance Benchmarks. ??
2013년 8월 2일 금요일
openstack compute
OpenStack Compute
The OpenStack cloud operating system enables enterprises and service providers to offer on-demand computing resources, by provisioning and managing large networks of virtual machines. Compute resources are accessible via APIs for developers building cloud applications and via web interfaces for administrators and users. The compute architecture is designed to scale horizontally on standard hardware, enabling the cloud economics companies have come to expect.
Flexible Architecture
OpenStack is architected to provide flexibility as you design your cloud, with no proprietary hardware or software requirements and the ability to integrate with legacy systems and third party technologies. It is designed to manage and automate pools of compute resources and can work with widely available virtualization technologies, as well as bare metal and high-performance computing (HPC) configurations.
Administrators often deploy OpenStack Compute using one of multiple supported hypervisors in a virtualized environment. KVM and XenServer are popular choices for hypervisor technology and recommended for most use cases. Linux container technology such as LXC is also supported for scenarios where users wish to minimize virtualization overhead and achieve greater efficiency and performance. In addition to different hypervisors, OpenStack supports ARM and alternative hardware architectures.
Administrators often deploy OpenStack Compute using one of multiple supported hypervisors in a virtualized environment. KVM and XenServer are popular choices for hypervisor technology and recommended for most use cases. Linux container technology such as LXC is also supported for scenarios where users wish to minimize virtualization overhead and achieve greater efficiency and performance. In addition to different hypervisors, OpenStack supports ARM and alternative hardware architectures.
Popular Use Cases
- Service providers offering an IaaS compute platform or services higher up the stack
- IT departments acting as cloud service providers for business units and project teams
- Processing big data with tools like Hadoop
- Scaling compute up and down to meet demand for web resources and applications
- High-performance computing (HPC) environments processing diverse and intensive workloads
2013년 8월 1일 목요일
mobile cloud architecture
mobile architecture requirements
1. mobile applications require omni channel delivery
- netflix to smart phones , tablets , laptops , TVs, game consoles
2. mobile applications require a mix of native code and Web view to blend capabilities
3. mobile applications require dynamic composition of RESTful services.
- Services remixed from Google, facebook , Twitter , Twilio , Spotify, ... everyday
4. mobile applications require use of open source SaaS and PaaS
- OpenMeap now available on Openshift , and AWS MarketPlace
5. moblie applications require elastic IaaS
- Instagram adding 1M new customers in 13 hours
6. mobile applications require super-agile development
- AWS being released every 11.6 seconds
7. mobile applications require context awareness
- google mytracks records your path , speed , distance and elevation while you walk ,run or bike
8. mobile applications require predictive analytics
- google now attempts to get you just the right information at just the right time by predicting your actions
9. mobile services must be immediate visual and simple
- apple siri takes the user's voice commands and input
10. mobile services must overcome last-mile connectivity bottleneck of 3G/4G
-microsoft's wherestore prototype allows caching data to mobile devices from clouds to reduce data unavailability and access latency
11. mobile services must provide the roles of remote control , routing , and data/computation offloading for wearables and connectables
-Nike + FuelBand tracks your daily acitivies and calorie burned
12. shared infrastructure and multitenancy : learn to share
13. elasticity and scalability : all you can eat
14. On-demand self-service with simple interface : get it your self
15. consumption based pricing : pay per use
16. cloud computing model drives revolutionary changes in how solutions are designed , built , delivered , sourced and managed
Mobile context computing architecture
1. mobile services must be contextual
2. User's mobile context is recognized via sensors
- GPS , NFC , gyroscopes, barometer, accelerometers , microbolometer, magnetometer , chemical sensor ,...
3. User's mobile context is :
- Situational : current time, location, attitude, environmental conditions, travel speeds..
- Preferential : historical personal decisions
- Attitudinal : feelings and emotions implied by actions and logistics
4. User's mobile context is stateful
5. manage states in clients subject to device capacities
- allows pipes and filters and broker architecture mixed with MVC architecture for mobile clients
- allows atomic services
- allows stateless , asynchronous communications with clouds
- allow load balancing and linear scale-out for IaaS
6. Use cloud to store and process the exploding amount of mobile contextual data, and to run predictive analytics.
1. mobile applications require omni channel delivery
- netflix to smart phones , tablets , laptops , TVs, game consoles
2. mobile applications require a mix of native code and Web view to blend capabilities
3. mobile applications require dynamic composition of RESTful services.
- Services remixed from Google, facebook , Twitter , Twilio , Spotify, ... everyday
4. mobile applications require use of open source SaaS and PaaS
- OpenMeap now available on Openshift , and AWS MarketPlace
5. moblie applications require elastic IaaS
- Instagram adding 1M new customers in 13 hours
6. mobile applications require super-agile development
- AWS being released every 11.6 seconds
7. mobile applications require context awareness
- google mytracks records your path , speed , distance and elevation while you walk ,run or bike
8. mobile applications require predictive analytics
- google now attempts to get you just the right information at just the right time by predicting your actions
9. mobile services must be immediate visual and simple
- apple siri takes the user's voice commands and input
10. mobile services must overcome last-mile connectivity bottleneck of 3G/4G
-microsoft's wherestore prototype allows caching data to mobile devices from clouds to reduce data unavailability and access latency
11. mobile services must provide the roles of remote control , routing , and data/computation offloading for wearables and connectables
-Nike + FuelBand tracks your daily acitivies and calorie burned
12. shared infrastructure and multitenancy : learn to share
13. elasticity and scalability : all you can eat
14. On-demand self-service with simple interface : get it your self
15. consumption based pricing : pay per use
16. cloud computing model drives revolutionary changes in how solutions are designed , built , delivered , sourced and managed
Mobile context computing architecture
1. mobile services must be contextual
2. User's mobile context is recognized via sensors
- GPS , NFC , gyroscopes, barometer, accelerometers , microbolometer, magnetometer , chemical sensor ,...
3. User's mobile context is :
- Situational : current time, location, attitude, environmental conditions, travel speeds..
- Preferential : historical personal decisions
- Attitudinal : feelings and emotions implied by actions and logistics
4. User's mobile context is stateful
5. manage states in clients subject to device capacities
- allows pipes and filters and broker architecture mixed with MVC architecture for mobile clients
- allows atomic services
- allows stateless , asynchronous communications with clouds
- allow load balancing and linear scale-out for IaaS
6. Use cloud to store and process the exploding amount of mobile contextual data, and to run predictive analytics.
2013년 7월 31일 수요일
Big data Architecture approach stretegy
We need new approach to the data
Traditional Approach
structured , logical , analytical
Data Warehouse : Transaction data
Internal App Data , Mainframe Data : Structured Repeatable Linear
OLTP System Data , ERP Data : Traditional Sources
New Approach
Creative , holistic thought , contextual
Hadoop Streaming Data : Web logs , URLs
Unstructured Exploratory Iterative ; Text Data : emails , chats , Social Data
New Sources : RFID , Sensor data , Network data
We don't have better algorithms than anyone else we just have more data
- peter Norvig Chief Scientist Google
Industrial Internet : Machine-based analytics : physics - based, deep domain expertise. automated , predictive
- Intelligent Machines : Connect the world's machines. facilities, fleets and networks with advanced sensors, controls and software application
- Advanced Analytics : combines the power of physics - based analytics, predictive algorithms, automation and deep domain expertise
- People at Work : Connecting People work or on the move , any time to support more intelligent design , operations, maintenance and higher sevice quality and safety
Industrial internet data loop :
1. Intelligence flows back into machines
2. Extraction and storage of proprietary machine data stream
3. Machine - based algorithms and data analysis
4. Remote and Centralized data visualization
5. Data Sharing with the right people and machines
It's necessary to include Big data in the formal DW Architecture
1. advanced analystics in the formal Dw Architecture
2. add Big data process architecture
3. expand real time process architecture
4. discover the new information through the search
Traditional Approach
structured , logical , analytical
Data Warehouse : Transaction data
Internal App Data , Mainframe Data : Structured Repeatable Linear
OLTP System Data , ERP Data : Traditional Sources
New Approach
Creative , holistic thought , contextual
Hadoop Streaming Data : Web logs , URLs
Unstructured Exploratory Iterative ; Text Data : emails , chats , Social Data
New Sources : RFID , Sensor data , Network data
We don't have better algorithms than anyone else we just have more data
- peter Norvig Chief Scientist Google
Industrial Internet : Machine-based analytics : physics - based, deep domain expertise. automated , predictive
- Intelligent Machines : Connect the world's machines. facilities, fleets and networks with advanced sensors, controls and software application
- Advanced Analytics : combines the power of physics - based analytics, predictive algorithms, automation and deep domain expertise
- People at Work : Connecting People work or on the move , any time to support more intelligent design , operations, maintenance and higher sevice quality and safety
Industrial internet data loop :
1. Intelligence flows back into machines
2. Extraction and storage of proprietary machine data stream
3. Machine - based algorithms and data analysis
4. Remote and Centralized data visualization
5. Data Sharing with the right people and machines
It's necessary to include Big data in the formal DW Architecture
1. advanced analystics in the formal Dw Architecture
2. add Big data process architecture
3. expand real time process architecture
4. discover the new information through the search
2013년 7월 24일 수요일
hadoop definitive guide ch 1.2.3
1. Apache Hadoop and the Hadoop Ecosystem
The Hadoop projects that are covered in this book are described briefly here:
Common
A set of components and interfaces for distributed filesystems and general I/O
(serialization, Java RPC, persistent data structures).
Avro
A serialization system for efficient, cross-language RPC, and persistent data
storage.
MapReduce
A distributed data processing model and execution environment that runs on large
clusters of commodity machines.
HDFS
A distributed filesystem that runs on large clusters of commodity machines.
Pig
A data flow language and execution environment for exploring very large datasets.
Pig runs on HDFS and MapReduce clusters.
Hive
A distributed data warehouse. Hive manages data stored in HDFS and provides a
query language based on SQL (and which is translated by the runtime engine to
MapReduce jobs) for querying the data.
HBase
A distributed, column-oriented database. HBase uses HDFS for its underlying
storage, and supports both batch-style computations using MapReduce and point
queries (random reads).
ZooKeeper
A distributed, highly available coordination service. ZooKeeper provides primitives
such as distributed locks that can be used for building distributed applications.
Sqoop
A tool for efficiently moving data between relational databases and HDFS.
2. map Reduce Data flow
Map tasks write their output to the local disk, not to HDFS. Why is this? Map output
is intermediate output: it’s processed by reduce tasks to produce the final output, and
once the job is complete the map output can be thrown away. So storing it in HDFS,
with replication, would be overkill. If the node running the map task fails before the
map output has been consumed by the reduce task, then Hadoop will automatically
rerun the map task on another node to re-create the map output.
Reduce tasks don’t have the advantage of data locality—the input to a single reduce
task is normally the output from all mappers. In the present example, we have a single
reduce task that is fed by all of the map tasks. Therefore, the sorted map outputs have
to be transferred across the network to the node where the reduce task is running, where
they are merged and then passed to the user-defined reduce function. The output of
the reduce is normally stored in HDFS for reliability. As explained in Chapter 3, for
each HDFS block of the reduce output, the first replica is stored on the local node, with
other replicas being stored on off-rack nodes. Thus, writing the reduce output does
consume network bandwidth, but only as much as a normal HDFS write pipeline
consumes.
3. Combiner Functions
Many MapReduce jobs are limited by the bandwidth available on the cluster, so it pays
to minimize the data transferred between map and reduce tasks. Hadoop allows the
user to specify a combiner function to be run on the map output—the combiner function’s
output forms the input to the reduce function.
4. The Design of HDFS
HDFS is a filesystem designed for storing very large files with streaming data access
patterns, running on clusters of commodity hardware
Very large files
Streaming data access
Commodity hardware
Low-latency data access
Lots of small files
Multiple writers, arbitrary file modifications
5. Namenodes and Datanodes
An HDFS cluster has two types of node operating in a master-worker pattern: a namenode (the master) and a number of datanodes (workers). The namenode manages the filesystem namespace. It maintains the filesystem tree and the metadata for all the files and directories in the tree. This information is stored persistently on the local disk in
the form of two files: the namespace image and the edit log. The namenode also knows
the datanodes on which all the blocks for a given file are located, however, it does
not store block locations persistently, since this information is reconstructed from
datanodes when the system starts.
A client accesses the filesystem on behalf of the user by communicating with the namenode and datanodes. The client presents a POSIX-like filesystem interface, so the user code does not need to know about the namenode and datanode to function.
Datanodes are the workhorses of the filesystem. They store and retrieve blocks when
they are told to (by clients or the namenode), and they report back to the namenode
periodically with lists of blocks that they are storing.
from hadoop definitive guide ch 1.2.3
The Hadoop projects that are covered in this book are described briefly here:
Common
A set of components and interfaces for distributed filesystems and general I/O
(serialization, Java RPC, persistent data structures).
Avro
A serialization system for efficient, cross-language RPC, and persistent data
storage.
MapReduce
A distributed data processing model and execution environment that runs on large
clusters of commodity machines.
HDFS
A distributed filesystem that runs on large clusters of commodity machines.
Pig
A data flow language and execution environment for exploring very large datasets.
Pig runs on HDFS and MapReduce clusters.
Hive
A distributed data warehouse. Hive manages data stored in HDFS and provides a
query language based on SQL (and which is translated by the runtime engine to
MapReduce jobs) for querying the data.
HBase
A distributed, column-oriented database. HBase uses HDFS for its underlying
storage, and supports both batch-style computations using MapReduce and point
queries (random reads).
ZooKeeper
A distributed, highly available coordination service. ZooKeeper provides primitives
such as distributed locks that can be used for building distributed applications.
Sqoop
A tool for efficiently moving data between relational databases and HDFS.
2. map Reduce Data flow
Map tasks write their output to the local disk, not to HDFS. Why is this? Map output
is intermediate output: it’s processed by reduce tasks to produce the final output, and
once the job is complete the map output can be thrown away. So storing it in HDFS,
with replication, would be overkill. If the node running the map task fails before the
map output has been consumed by the reduce task, then Hadoop will automatically
rerun the map task on another node to re-create the map output.
Reduce tasks don’t have the advantage of data locality—the input to a single reduce
task is normally the output from all mappers. In the present example, we have a single
reduce task that is fed by all of the map tasks. Therefore, the sorted map outputs have
to be transferred across the network to the node where the reduce task is running, where
they are merged and then passed to the user-defined reduce function. The output of
the reduce is normally stored in HDFS for reliability. As explained in Chapter 3, for
each HDFS block of the reduce output, the first replica is stored on the local node, with
other replicas being stored on off-rack nodes. Thus, writing the reduce output does
consume network bandwidth, but only as much as a normal HDFS write pipeline
consumes.
3. Combiner Functions
Many MapReduce jobs are limited by the bandwidth available on the cluster, so it pays
to minimize the data transferred between map and reduce tasks. Hadoop allows the
user to specify a combiner function to be run on the map output—the combiner function’s
output forms the input to the reduce function.
4. The Design of HDFS
HDFS is a filesystem designed for storing very large files with streaming data access
patterns, running on clusters of commodity hardware
Very large files
Streaming data access
Commodity hardware
Low-latency data access
Lots of small files
Multiple writers, arbitrary file modifications
5. Namenodes and Datanodes
An HDFS cluster has two types of node operating in a master-worker pattern: a namenode (the master) and a number of datanodes (workers). The namenode manages the filesystem namespace. It maintains the filesystem tree and the metadata for all the files and directories in the tree. This information is stored persistently on the local disk in
the form of two files: the namespace image and the edit log. The namenode also knows
the datanodes on which all the blocks for a given file are located, however, it does
not store block locations persistently, since this information is reconstructed from
datanodes when the system starts.
A client accesses the filesystem on behalf of the user by communicating with the namenode and datanodes. The client presents a POSIX-like filesystem interface, so the user code does not need to know about the namenode and datanode to function.
Datanodes are the workhorses of the filesystem. They store and retrieve blocks when
they are told to (by clients or the namenode), and they report back to the namenode
periodically with lists of blocks that they are storing.
from hadoop definitive guide ch 1.2.3
2012년 11월 20일 화요일
Machine learning in action
Recently I read the machine learning in action
Until Now I read often about classification !!
Fortunately I already read related materials in the Algorithms of intelligent Web , Programming Collective intelligence and Mahout in Action
That books also introduce the classification !! I think I am familiar with that subject
Data mining - Practical Machine Learning Tools and Techniques is also contain basic concepts
Classification
Clustering
Machine Learnning
Data Mining
Big Data
Hadoop
Search Algorithms
Text Mining
What am i looking for something ?
And I study python again !!
Until Now I read often about classification !!
Fortunately I already read related materials in the Algorithms of intelligent Web , Programming Collective intelligence and Mahout in Action
That books also introduce the classification !! I think I am familiar with that subject
Data mining - Practical Machine Learning Tools and Techniques is also contain basic concepts
Classification
Clustering
Machine Learnning
Data Mining
Big Data
Hadoop
Search Algorithms
Text Mining
What am i looking for something ?
And I study python again !!
피드 구독하기:
글 (Atom)