Commit e3bc8e06 authored by yushiqian's avatar yushiqian

copy pig lab

parent 8b41320a
This diff is collapsed.
# Airline Data Analysis with Pig
+ This exercise is inspired by http://www.datadr.org/doc/airline.html
+ Full information on datasets (optional datasets), and general documentation available here: http://stat-computing.org/dataexpo/2009/
Before we start, here's a description of the dataset "schema". We will work on data that can be downloaded from here: http://stat-computing.org/dataexpo/2009/the-data.html
Note that there is a single CSV file per year, hence the first field below is somehow redundant, although you can imagine to concatenate all files and work on them as a whole (which by the way would make sense when using Hadoop MapReduce / Pig). In summary, there are 29 fields which provide enough information to build Pig scripts that cover Queries 1-5. For the advanced analysis subsection, you need other data, which can be downloaded from the links below.
```
1 Year 1987-2008
2 Month 1-12
3 DayofMonth 1-31
4 DayOfWeek 1 (Monday) - 7 (Sunday)
5 DepTime actual departure time (local, hhmm)
6 CRSDepTime scheduled departure time (local, hhmm)
7 ArrTime actual arrival time (local, hhmm)
8 CRSArrTime scheduled arrival time (local, hhmm)
9 UniqueCarrier unique carrier code
10 FlightNum flight number
11 TailNum plane tail number
12 ActualElapsedTime in minutes
13 CRSElapsedTime in minutes
14 AirTime in minutes
15 ArrDelay arrival delay, in minutes
16 DepDelay departure delay, in minutes
17 Origin origin IATA airport code
18 Dest destination IATA airport code
19 Distance in miles
20 TaxiIn taxi in time, in minutes
21 TaxiOut taxi out time in minutes
22 Cancelled was the flight cancelled?
23 CancellationCode reason for cancellation (A = carrier, B = weather, C = NAS, D = security)
24 Diverted 1 = yes, 0 = no
25 CarrierDelay in minutes
26 WeatherDelay in minutes
27 NASDelay in minutes
28 SecurityDelay in minutes
29 LateAircraftDelay in minutes
```
Other sources of data come from here: http://stat-computing.org/dataexpo/2009/supplemental-data.html. Precisely, we are interested in:
+ Airport IATA Codes to City names and Coordinates mapping: http://stat-computing.org/dataexpo/2009/airports.csv
+ Carrier codes to Full name mapping: http://stat-computing.org/dataexpo/2009/carriers.csv
+ Information about individual planes: http://stat-computing.org/dataexpo/2009/plane-data.csv
+ Weather information: http://www.wunderground.com/weather/api/. You can subscribe for free to the developers API and obtain (at a limited rate) hystorical weather information in many different formats. Also, to get an idea of the kind of information is available, you can use this link: http://www.wunderground.com/history/
## The Data:
For Eurecom students, we have put some csv (comma separate values) files in HDFS for you. They are located in the HDFS directory: ```/laboratory/airlines/``` and they account for four years of data, 2005 to 2008. You can work on individual years, or feed the entire directory to your jobs such that you can process all years.
## Exercises:
In the following, we propose a series of exercises in the form of Queries.
### *Query 1:* Top 20 airports by total volume of flights
What are the busiest airports by total flight traffic. JFK will feature, but what are the others? For each airport code compute the number of inbound, outbound and all flights. Variation on the theme: compute the above by day, week, month, and over the years.
### *Query 2:* Carrier Popularity
Some carriers come and go, others demonstrate regular growth. Compute the (log base 10) volume -- total flights -- over each year, by carrier. The carriers are ranked by their median volume (over the 4 year span).
### *Query 3:* Proportion of Flights Delayed
A flight is delayed if the delay is greater than 15 minutes. Compute the fraction of delayed flights per different time granularities (hour, day, week, month, year).
### *Query 4:* Carrier Delays
Is there a difference in carrier delays? Compute the proportion of delayed flights by carrier, ranked by carrier, at different time granularities (hour, day, week, month year). Again, a flight is delayed if the delay is greater than 15 minutes.
### *Query 5:* Busy Routes
Which routes are the busiest? A simple first approach is to create a frequency table for the unordered pair (i,j) where i and j are distinct airport codes.
## Optional data analysis tasks
Note that the following "queries" are somehow more involved than the preceding ones. They require a more elaborated approach and in some cases they require additional data to be downloaded (see sources above) and copied into HDFS.
+ When is the best time of day/day of week/time of year to fly to minimize delays?
+ Do older planes suffer more delays?
+ How does the number of people flying between different locations change over time?
+ How well does weather predict plane delays?
+ Can you detect cascading failures as delays in one airport create delays in others? Are there critical links in the system?
This diff is collapsed.
Error before Pig is launched
----------------------------
ERROR 2997: Encountered IOException. File sample-solutions/WORD_COUNT/word_count.pig does not exist
java.io.FileNotFoundException: File sample-solutions/WORD_COUNT/word_count.pig does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:524)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:737)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:514)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:744)
at org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:688)
at org.apache.pig.Main.run(Main.java:549)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
================================================================================
Pig Stack Trace
---------------
ERROR 2244: Job failed, hadoop does not return any error message
org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:148)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
================================================================================
Pig Stack Trace
---------------
ERROR 6000:
<file ./sample-solutions/WORD_COUNT/word_count.pig, line 15, column 0> Output Location Validation Failed for: 'file:///home/group37/CLOUDS-LAB/labs/pig-lab/local-output/WORD_COUNT More info to follow:
Output directory file:/home/group37/CLOUDS-LAB/labs/pig-lab/local-output/WORD_COUNT already exists
org.apache.pig.impl.plan.VisitorException: ERROR 6000:
<file ./sample-solutions/WORD_COUNT/word_count.pig, line 15, column 0> Output Location Validation Failed for: 'file:///home/group37/CLOUDS-LAB/labs/pig-lab/local-output/WORD_COUNT More info to follow:
Output directory file:/home/group37/CLOUDS-LAB/labs/pig-lab/local-output/WORD_COUNT already exists
at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311)
at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1305)
at org.apache.pig.PigServer.execute(PigServer.java:1297)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/home/group37/CLOUDS-LAB/labs/pig-lab/local-output/WORD_COUNT already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
... 27 more
================================================================================
Pig Stack Trace
---------------
ERROR 2997: Encountered IOException. org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1426847610111_2382' doesn't exist in RM.
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
java.io.IOException: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1426847610111_2382' doesn't exist in RM.
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:348)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:553)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:582)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:580)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:629)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:150)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:428)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.execute(PigServer.java:1297)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
================================================================================
Pig Stack Trace
---------------
ERROR 2997: Encountered IOException. org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1426847610111_2407' doesn't exist in RM.
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
java.io.IOException: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1426847610111_2407' doesn't exist in RM.
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:348)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:553)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:582)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:580)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:629)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:150)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:428)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.execute(PigServer.java:1297)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
================================================================================
Pig Stack Trace
---------------
ERROR 6000:
<file ./sample-solutions/WORD_COUNT/word_count.pig, line 15, column 0> Output Location Validation Failed for: 'hdfs://bigdoop-1.vms.bigfoot.eurecom.fr:8020/user/group37/local-output/WORD_COUNT More info to follow:
Output directory hdfs://bigdoop-1.vms.bigfoot.eurecom.fr:8020/user/group37/local-output/WORD_COUNT already exists
org.apache.pig.impl.plan.VisitorException: ERROR 6000:
<file ./sample-solutions/WORD_COUNT/word_count.pig, line 15, column 0> Output Location Validation Failed for: 'hdfs://bigdoop-1.vms.bigfoot.eurecom.fr:8020/user/group37/local-output/WORD_COUNT More info to follow:
Output directory hdfs://bigdoop-1.vms.bigfoot.eurecom.fr:8020/user/group37/local-output/WORD_COUNT already exists
at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311)
at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1305)
at org.apache.pig.PigServer.execute(PigServer.java:1297)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://bigdoop-1.vms.bigfoot.eurecom.fr:8020/user/group37/local-output/WORD_COUNT already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
... 27 more
================================================================================
Pig Stack Trace
---------------
ERROR 1000: Error during parsing. Encountered " <IDENTIFIER> "exit "" at line 11, column 1.
Was expecting one of:
<EOF>
"cat" ...
"clear" ...
"fs" ...
"sh" ...
"cd" ...
"cp" ...
"copyFromLocal" ...
"copyToLocal" ...
"dump" ...
"\\d" ...
"describe" ...
"\\de" ...
"aliases" ...
"explain" ...
"\\e" ...
"help" ...
"history" ...
"kill" ...
"ls" ...
"mv" ...
"mkdir" ...
"pwd" ...
"quit" ...
"\\q" ...
"register" ...
"rm" ...
"rmf" ...
"set" ...
"illustrate" ...
"\\i" ...
"run" ...
"exec" ...
"scriptDone" ...
"" ...
"" ...
<EOL> ...
";" ...
org.apache.pig.tools.pigscript.parser.ParseException: Encountered " <IDENTIFIER> "exit "" at line 11, column 1.
Was expecting one of:
<EOF>
"cat" ...
"clear" ...
"fs" ...
"sh" ...
"cd" ...
"cp" ...
"copyFromLocal" ...
"copyToLocal" ...
"dump" ...
"\\d" ...
"describe" ...
"\\de" ...
"aliases" ...
"explain" ...
"\\e" ...
"help" ...
"history" ...
"kill" ...
"ls" ...
"mv" ...
"mkdir" ...
"pwd" ...
"quit" ...
"\\q" ...
"register" ...
"rm" ...
"rmf" ...
"set" ...
"illustrate" ...
"\\i" ...
"run" ...
"exec" ...
"scriptDone" ...
"" ...
"" ...
<EOL> ...
";" ...
at org.apache.pig.tools.pigscript.parser.PigScriptParser.generateParseException(PigScriptParser.java:1428)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.handle_invalid_command(PigScriptParser.java:1236)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:710)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:541)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
================================================================================
Name Description
1 Year 1987-2008
2 Month 1-12
3 DayofMonth 1-31
4 DayOfWeek 1 (Monday) - 7 (Sunday)
5 DepTime actual departure time (local, hhmm)
6 CRSDepTime scheduled departure time (local, hhmm)
7 ArrTime actual arrival time (local, hhmm)
8 CRSArrTime scheduled arrival time (local, hhmm)
9 UniqueCarrier unique carrier code
10 FlightNum flight number
11 TailNum plane tail number
12 ActualElapsedTime in minutes
13 CRSElapsedTime in minutes
14 AirTime in minutes
15 ArrDelay arrival delay, in minutes
16 DepDelay departure delay, in minutes
17 Origin origin IATA airport code
18 Dest destination IATA airport code
19 Distance in miles
20 TaxiIn taxi in time, in minutes
21 TaxiOut taxi out time in minutes
22 Cancelled was the flight cancelled?
23 CancellationCode reason for cancellation (A = carrier, B = weather, C = NAS, D = security)
24 Diverted 1 = yes, 0 = no
25 CarrierDelay in minutes
26 WeatherDelay in minutes
27 NASDelay in minutes
28 SecurityDelay in minutes
29 LateAircraftDelay in minutes
This source diff could not be displayed because it is too large. You can view the blob instead.
This diff is collapsed.
2008,1,3,4,754,735,1002,1000,WN,3231,N772SW,128,145,113,2,19,IAD,TPA,810,5,10,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,628,620,804,750,WN,448,N428WN,96,90,76,14,8,IND,BWI,515,3,17,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,754,735,1002,1000,WN,3231,N772SW,128,145,113,2,19,IAD,TPA,810,5,10,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,628,620,804,750,WN,448,N428WN,96,90,76,14,8,IND,BWI,515,3,17,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,926,930,1054,1100,WN,1746,N612SW,88,90,78,-6,-4,IND,BWI,515,3,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1829,1755,1959,1925,WN,3920,N464WN,90,90,77,34,34,IND,BWI,515,3,10,0,,0,2,0,0,0,32
2008,1,3,4,1940,1915,2121,2110,WN,378,N726SW,101,115,87,11,25,IND,JAX,688,4,10,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1937,1830,2037,1940,WN,509,N763SW,240,250,230,57,67,IND,LAS,1591,3,7,0,,0,10,0,0,0,47
2008,1,3,4,1039,1040,1132,1150,WN,535,N428WN,233,250,219,-18,-1,IND,LAS,1591,7,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,617,615,652,650,WN,11,N689SW,95,95,70,2,2,IND,MCI,451,6,19,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1620,1620,1639,1655,WN,810,N648SW,79,95,70,-16,0,IND,MCI,451,3,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,706,700,916,915,WN,100,N690SW,130,135,106,1,6,IND,MCO,828,5,19,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1644,1510,1845,1725,WN,1333,N334SW,121,135,107,80,94,IND,MCO,828,6,8,0,,0,8,0,0,0,72
2008,1,3,4,1426,1430,1426,1425,WN,829,N476WN,60,55,39,1,-4,IND,MDW,162,9,12,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,715,715,720,710,WN,1016,N765SW,65,55,37,10,0,IND,MDW,162,7,21,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1702,1700,1651,1655,WN,1827,N420WN,49,55,35,-4,2,IND,MDW,162,4,10,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1029,1020,1021,1010,WN,2272,N263WN,52,50,37,11,9,IND,MDW,162,6,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1452,1425,1640,1625,WN,675,N286WN,228,240,213,15,27,IND,PHX,1489,7,8,0,,0,3,0,0,0,12
2008,1,3,4,754,745,940,955,WN,1144,N778SW,226,250,205,-15,9,IND,PHX,1489,5,16,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1323,1255,1526,1510,WN,4,N674AA,123,135,110,16,28,IND,TPA,838,4,9,0,,0,0,0,0,0,16
2008,1,3,4,1416,1325,1512,1435,WN,54,N643SW,56,70,49,37,51,ISP,BWI,220,2,5,0,,0,12,0,0,0,25
2008,1,3,4,706,705,807,810,WN,68,N497WN,61,65,51,-3,1,ISP,BWI,220,3,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1657,1625,1754,1735,WN,623,N724SW,57,70,47,19,32,ISP,BWI,220,5,5,0,,0,7,0,0,0,12
2008,1,3,4,1900,1840,1956,1950,WN,717,N786SW,56,70,49,6,20,ISP,BWI,220,2,5,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1039,1030,1133,1140,WN,1244,N714CB,54,70,47,-7,9,ISP,BWI,220,2,5,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,801,800,902,910,WN,2101,N222WN,61,70,53,-8,1,ISP,BWI,220,3,5,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1520,1455,1619,1605,WN,2553,N394SW,59,70,50,14,25,ISP,BWI,220,2,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1422,1255,1657,1610,WN,188,N215WN,155,195,143,47,87,ISP,FLL,1093,6,6,0,,0,40,0,0,0,7
2008,1,3,4,1954,1925,2239,2235,WN,1754,N243WN,165,190,155,4,29,ISP,FLL,1093,3,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,636,635,921,945,WN,2275,N454WN,165,190,147,-24,1,ISP,FLL,1093,5,13,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,734,730,958,1020,WN,550,N712SW,324,350,314,-22,4,ISP,LAS,2283,2,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,2107,1945,2334,2230,WN,362,N798SW,147,165,134,64,82,ISP,MCO,972,6,7,0,,0,5,0,0,0,59
2008,1,3,4,1008,1005,1234,1255,WN,543,N736SA,146,170,135,-21,3,ISP,MCO,972,5,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,712,710,953,1000,WN,1112,N795SW,161,170,142,-7,2,ISP,MCO,972,5,14,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1312,1300,1546,1550,WN,1397,N247WN,154,170,140,-4,12,ISP,MCO,972,7,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1449,1430,1715,1720,WN,3398,N707SA,146,170,134,-5,19,ISP,MCO,972,6,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1634,1555,1859,1845,WN,3480,N443WN,145,170,134,14,39,ISP,MCO,972,5,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,831,830,935,955,WN,300,N753SW,124,145,112,-20,1,ISP,MDW,765,5,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1812,1650,1927,1815,WN,422,N779SW,135,145,118,72,82,ISP,MDW,765,6,11,0,,0,3,0,0,0,69
2008,1,3,4,1127,1105,1235,1230,WN,1837,N704SW,128,145,114,5,22,ISP,MDW,765,9,5,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1424,1355,1531,1520,WN,2871,N709SW,127,145,113,11,29,ISP,MDW,765,8,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1326,1230,1559,1530,WN,1056,N459WN,153,180,143,29,56,ISP,PBI,1052,5,5,0,,0,0,0,0,0,29
2008,1,3,4,1749,1725,2019,2030,WN,2175,N621SW,150,185,138,-11,24,ISP,PBI,1052,4,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,726,720,958,1020,WN,3319,N206WN,152,180,140,-22,6,ISP,PBI,1052,4,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,646,640,929,955,WN,3667,N280WN,163,195,151,-26,6,ISP,RSW,1101,3,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1153,1140,1428,1440,WN,2006,N241WN,155,180,143,-12,13,ISP,TPA,1034,4,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1528,1510,1802,1810,WN,3858,N200WN,154,180,144,-8,18,ISP,TPA,1034,4,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,634,635,907,935,WN,3928,N459WN,153,180,142,-28,-1,ISP,TPA,1034,3,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,831,830,1148,1140,WN,534,N286WN,137,130,123,8,1,JAN,BWI,888,5,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1450,1435,1806,1745,WN,3244,N475WN,136,130,121,21,15,JAN,BWI,888,7,8,0,,0,0,0,6,0,15
2008,1,3,4,2245,1730,2354,1850,WN,186,N792SW,69,80,59,304,315,JAN,HOU,359,3,7,0,,0,282,0,0,0,22
2008,1,3,4,615,615,724,735,WN,971,N202WN,69,80,60,-11,0,JAN,HOU,359,2,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1150,1145,1303,1305,WN,2124,N646SW,73,80,63,-2,5,JAN,HOU,359,3,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,2025,1940,2135,2100,WN,3154,N252WN,70,80,60,35,45,JAN,HOU,359,3,7,0,,0,26,0,0,0,9
2008,1,3,4,1038,945,1314,1225,WN,1035,N346SW,96,100,81,49,53,JAN,MCO,587,8,7,0,,0,7,0,0,0,42
2008,1,3,4,1900,1850,2123,2045,WN,205,N299WN,143,115,97,38,10,JAN,MDW,666,40,6,0,,0,1,0,28,0,9
2008,1,3,4,700,700,851,900,WN,449,N528SW,111,120,99,-9,0,JAN,MDW,666,6,6,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,948,925,959,940,WN,3430,N487WN,71,75,59,19,23,JAX,BHM,365,3,9,0,,0,0,0,0,0,19
2008,1,3,4,646,620,725,655,WN,1580,N243WN,99,95,77,30,26,JAX,BNA,484,6,16,0,,0,26,0,4,0,0
2008,1,3,4,1110,1040,1136,1110,WN,2195,N479WN,86,90,72,26,30,JAX,BNA,484,5,9,0,,0,0,0,0,10,16
2008,1,3,4,1535,1535,1603,1610,WN,2804,N255WN,88,95,74,-7,0,JAX,BNA,484,5,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1919,1915,1942,1950,WN,3428,N215WN,83,95,71,-8,4,JAX,BNA,484,4,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1053,1055,1245,1240,WN,433,N264LV,112,105,96,5,-2,JAX,BWI,663,7,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1433,1440,1623,1625,WN,1331,N714CB,110,105,95,-2,-7,JAX,BWI,663,2,13,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,2015,2010,2158,2155,WN,3504,N436WN,103,105,91,3,5,JAX,BWI,663,5,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,2139,2130,2244,2240,WN,378,N726SW,65,70,54,4,9,JAX,FLL,318,3,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1500,1500,1602,1615,WN,640,N399WN,62,75,49,-13,0,JAX,FLL,318,5,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,850,850,1000,1000,WN,1396,N387SW,70,70,51,0,0,JAX,FLL,318,10,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,646,645,752,755,WN,2189,N405WN,66,70,46,-3,1,JAX,FLL,318,13,7,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1221,1220,1328,1330,WN,3312,N685SW,67,70,54,-2,1,JAX,FLL,318,5,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1738,1730,1841,1840,WN,3948,N467WN,63,70,49,1,8,JAX,FLL,318,6,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1813,1735,1936,1905,WN,54,N643SW,143,150,125,31,38,JAX,HOU,816,6,12,0,,0,11,0,0,0,20
2008,1,3,4,802,750,1001,955,WN,2272,N263WN,119,125,104,6,12,JAX,IND,688,7,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1820,1825,1946,1955,WN,549,N363SW,86,90,75,-9,-5,JAX,ORF,543,3,8,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,821,820,953,945,WN,3604,N257WN,92,85,80,8,1,JAX,ORF,543,3,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1734,1650,1941,1905,WN,23,N521SW,127,135,113,36,44,JAX,PHL,742,4,10,0,,0,3,0,0,0,33
2008,1,3,4,712,700,926,915,WN,1232,N663SW,134,135,120,11,12,JAX,PHL,742,5,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1318,1310,1410,1400,WN,977,N376SW,52,50,39,10,8,JAX,TPA,180,4,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,958,900,1052,950,WN,1574,N791SW,54,50,36,62,58,JAX,TPA,180,4,14,0,,0,0,0,4,0,58
2008,1,3,4,1859,1850,1950,1945,WN,2019,N392SW,51,55,38,5,9,JAX,TPA,180,4,9,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1538,1445,1753,1710,WN,500,N799SW,75,85,60,43,53,LAS,ABQ,487,4,11,0,,0,15,0,0,0,28
2008,1,3,4,933,935,1151,1200,WN,778,N607SW,78,85,63,-9,-2,LAS,ABQ,487,5,10,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,2248,2125,102,2345,WN,890,N618WN,74,80,60,77,83,LAS,ABQ,487,4,10,0,,0,7,0,0,0,70
2008,1,3,4,1327,1230,1550,1500,WN,1171,N682SW,83,90,65,50,57,LAS,ABQ,487,3,15,0,,0,50,0,0,0,0
2008,1,3,4,624,625,846,850,WN,1320,N456WN,82,85,61,-4,-1,LAS,ABQ,487,4,17,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1614,1600,1833,1825,WN,1925,N509SW,79,85,60,8,14,LAS,ABQ,487,4,15,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1917,1915,2136,2140,WN,2457,N293,79,85,60,-4,2,LAS,ABQ,487,5,14,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1832,1655,148,30,WN,302,N473WN,256,275,243,78,97,LAS,ALB,2237,3,10,0,,0,8,0,0,0,70
2008,1,3,4,1229,1155,1633,1555,WN,1079,N351SW,124,120,91,38,34,LAS,AMA,758,5,28,0,,0,3,0,4,0,31
2008,1,3,4,1256,1240,1724,1720,WN,155,N238WN,148,160,131,4,16,LAS,AUS,1090,3,14,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,2118,2015,144,50,WN,2021,N499WN,146,155,127,54,63,LAS,AUS,1090,5,14,0,,0,23,0,0,0,31
2008,1,3,4,905,850,1334,1330,WN,3222,N309SW,149,160,135,4,15,LAS,AUS,1090,4,10,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1739,1640,114,25,WN,1018,N245WN,275,285,253,49,59,LAS,BDL,2298,9,13,0,,0,19,0,0,0,30
2008,1,3,4,906,905,1426,1430,WN,3948,N467WN,200,205,183,-4,1,LAS,BHM,1618,2,15,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,816,815,1339,1340,WN,249,N256WN,203,205,185,-1,1,LAS,BNA,1588,4,14,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1325,1240,1841,1810,WN,419,N275WN,196,210,178,31,45,LAS,BNA,1588,6,12,0,,0,2,0,0,0,29
2008,1,3,4,1506,1440,2030,2010,WN,2032,N271WN,204,210,183,20,26,LAS,BNA,1588,6,15,0,,0,18,0,0,0,2
2008,1,3,4,2039,1930,155,55,WN,3940,N434WN,196,205,177,60,69,LAS,BNA,1588,5,14,0,,0,0,0,22,0,38
2008,1,3,4,924,920,1209,1210,WN,71,N312SW,105,110,84,-1,4,LAS,BOI,520,3,18,0,,0,NA,NA,NA,NA,NA
2008,1,3,4,1611,1535,1849,1825,WN,538,N619SW,98,110,73,24,36,LAS,BOI,520,3,22,0,,0,9,0,0,0,15
2008,1,3,4,1824,1715,117,25,WN,2383,N290WN,233,250,221,52,69,LAS,BUF,1987,2,10,0,,0,48,0,0,0,4
2008,1,3,4,826,825,930,925,WN,136,N493WN,64,60,46,5,1,LAS,BUR,223,2,16,0,,0,NA,NA,NA,NA,NA
This diff is collapsed.
1238337901 48622 80 189870634 3241471387 4 6 16 101 101 5454 143420 141400
1238337901 64812 80 609698120 2446585429 4 6 16 45 45 2970 66052 65152
1238337901 36221 62134 609697803 3241471387 4 6 16 131 131 8646 186020 183400
1238337901 80 2588 2908670611 877875549 4 6 16 9 9 582 456 276
1238337901 59424 80 609697931 2446577108 4 6 16 2 2 132 104 64
1238337901 65535 65535 3241468042 837367666 2 1 255 25 25 1050 1600 1100
1238337901 61921 80 3491926332 2908670611 4 6 16 3 3 198 3052 2992
1238337901 57160 80 19497852 3241471386 4 6 17 1 1 54 40 20
1238337901 61921 80 3491926332 2908670611 4 6 24 1 1 66 132 112
1238337901 61921 80 3491926332 2908670611 4 6 17 1 1 66 52 32
1238337901 3783 80 983681490 2908670611 4 6 16 5 5 270 6040 5940
1238337901 2099 80 1264921632 2908670611 4 6 16 11 11 594 15400 15180
1238337901 80 58067 613331040 2908624170 4 6 16 25 25 1650 1300 800
1238337901 59424 80 609697931 2446577108 4 6 24 1 1 66 211 191
1238337901 59424 80 609697931 2446577108 4 6 17 1 1 66 52 32
1238337901 40881 54478 3241469417 208551314 4 6 16 11 11 726 572 352
1238337901 64930 1341 2502046810 2512113175 4 6 16 15 15 922 712 412
1238337901 64812 80 609698120 2446585429 4 6 24 2 2 132 1816 1776
1238337901 1798 6664 2820868137 3295580861 4 6 24 1 1 54 110 90
1238337901 49391 80 839950143 2908670611 4 6 16 17 17 918 24140 23800
1238337901 9106 51151 955783554 2908677624 4 6 16 9 9 486 360 180
1238337901 80 2827 3241471387 202636863 4 6 16 37 37 1998 1480 740
1238337901 53 54664 3413010761 2728877993 3 17 255 1 1 54 70 50
1238337901 18751 3124 3670878926 3241470141 4 6 24 1 1 54 294 274
1238337901 80 60370 3241471386 714401720 4 6 16 11 11 594 440 220
1238337901 63016 80 807057262 2908670611 4 6 16 20 20 1080 30000 29600
1238337901 1728 80 727430293 2908670611 4 6 16 3 3 162 4500 4440
1238337901 34070 56232 296541965 2418833098 3 17 255 12 12 504 13272 13032
1238337901 3124 18732 3241470141 3670878926 4 6 16 2 2 132 104 64
1238337901 65535 65535 3241470129 3397308854 2 1 255 1 1 42 124 104
1238337901 18732 3124 3670878926 3241470141 4 6 16 2 2 108 2960 2920
1238337901 1377 80 841278366 3241470593 4 6 16 6 6 324 8880 8760
1238337901 58411 80 1387436594 3241471386 4 6 16 117 117 6318 174564 172224
1238337901 80 4259 2908670611 912474870 4 6 16 13 13 702 520 260
1238337901 35605 80 2987784296 3241471386 4 6 16 30 30 1620 45000 44400
1238337901 80 1364 2908670611 685806622 4 6 16 3 3 214 172 112
1238337901 62140 1293 3224014473 140469055 3 17 255 1 1 42 183 163
1238337901 52817 8080 989879316 2908670686 4 6 16 111 111 5994 166500 164280
1238337901 80 2626 3241471386 3719125872 4 6 16 6 6 324 240 120
1238337901 80 52363 2761615119 2510459665 4 6 16 96 96 5184 3840 1920
1238337901 10053 53 2760918353 3241470141 3 17 255 1 1 54 418 398
1238337901 1194 58451 2908672721 1867388188 3 17 255 363 363 15246 39907 32647
1238337901 80 58411 3241471386 1387436594 4 6 16 57 57 3078 2280 1140
1238337901 58067 80 2908624170 613331040 4 6 16 56 56 3696 84000 82880
1238337901 46265 80 1397846996 2908670686 4 6 24 3 3 162 4500 4440
1238337901 49524 80 271298300 2908670611 4 6 16 9 9 486 13500 13320
1238337901 18751 3124 3670878926 3241470141 4 6 17 1 1 54 40 20
1238337901 29300 12340 2908623326 3233526254 3 17 255 44 44 1848 2718 1838
1238337901 53985 80 3495491322 3241471386 4 6 16 46 46 2484 65320 64400
1238337901 80 3513 3241467907 1393163140 4 6 16 45 45 2970 2340 1440
1238337901 3124 3530 3241470129 3508331649 4 6 16 1 1 54 40 20
1238337901 3124 3530 3241470129 3508331649 4 6 17 1 1 54 40 20
1238337901 65535 65535 837367666 3241468042 2 1 255 47 47 1974 3008 2068
1238337901 25 6812 2510424753 3696207645 4 6 2 1 1 62 48 28
1238337901 41728 41728 3241470140 2006258665 3 17 255 3 3 126 814 754
1238337901 8080 52817 2908670686 989879316 4 6 16 58 58 3132 2320 1160
1238337901 65535 65535 679519382 2234296832 2 1 255 1 1 70 56 36
1238337901 13657 11739 2908623106 1000524350 3 17 255 31 31 1302 1643 1023
1238337901 80 11181 3241467907 3666742197 4 6 16 7 7 390 292 152
1238337901 41728 41728 2006258665 3241470140 3 17 255 3 3 126 1292 1232
1238337901 53238 53 2887647489 3230195100 3 17 255 1 1 54 459 439
1238337901 65535 65535 1377867173 2234307738 2 1 255 1 1 70 56 36
1238337901 80 51824 2908670611 831898697 4 6 16 6 6 324 240 120
1238337901 65535 65535 3241470129 3328466941 2 1 255 1 1 70 56 36
1238337901 62345 46372 2418843024 1389649717 3 17 255 11 11 462 605 385
1238337901 3801 80 1036706626 2908670611 4 6 16 11 11 594 15400 15180
1238337901 65535 65535 2908670672 2061567840 2 1 255 1 1 70 56 36
1238337901 53 57705 3413010761 142946216 3 17 255 1 1 54 58 38
1238337901 4520 80 147040931 3241471386 4 6 16 10 10 540 14000 13800
1238337901 4669 25 46731798 2418807062 4 6 24 2 2 108 237 197
1238337901 2249 80 3658832411 3241471387 4 6 16 106 106 5724 150520 148400
1238337901 3658 3658 3590469441 2908622921 3 17 255 9 9 378 1012 832
1238337901 1434 2639 2923383954 1413716685 3 17 255 1 1 42 404 384
1238337901 80 49931 3241470593 896350321 4 6 16 10 10 540 400 200
1238337901 6517 8890 2234255650 3728597840 4 6 18 1 1 62 48 28
1238337901 3124 3556 3241470129 3508331649 4 6 2 1 1 62 48 28
1238337901 11739 13657 1000524350 2908623106 3 17 255 32 32 1344 34848 34208
1238337901 80 27201 3241467907 3666742197 4 6 16 12 12 648 480 240
1238337901 61110 53 2887647489 3230195100 3 17 255 1 1 54 463 443
1238337901 54505 80 3593411485 3241467907 4 6 16 18 18 1188 25560 25200
1238337901 49931 80 896350321 3241470593 4 6 16 21 21 1134 31500 31080
1238337901 53 1026 3413010761 3271255311 3 17 255 1 1 54 52 32
1238337901 2236 80 26126404 3241471387 4 6 16 30 30 1620 42600 42000
1238337901 36924 80 2234275547 3665874141 4 6 18 1 1 62 48 28
1238337901 80 58424 1013915290 2418820813 4 6 16 3 3 198 156 96
1238337901 65535 65535 3361087795 3241470141 2 1 255 4 4 168 496 416
1238337901 65535 65535 4294967295 4294967295 2 255 255 154 0 13042 0 0
1238337901 53 51018 3413010761 1011144641 3 17 255 1 1 54 53 33
1238337901 80 58424 1013915290 2418820813 4 6 24 1 1 66 624 604
1238337901 80 55843 2908670611 1000530837 4 6 16 4 4 216 160 80
1238337901 52247 53 2887647489 3230195100 3 17 255 1 1 54 515 495
1238337901 55499 80 766440910 2418811652 4 6 16 200 200 13200 299896 295896
1238337901 65535 65535 1401532070 3417261585 2 41 255 3 3 282 3900 3840
1238337901 1267 3124 764114878 3241470129 4 6 16 1 1 54 40 20
1238337901 13312 46689 3241467907 47236760 4 6 16 30 30 1980 1560 960
1238337901 16785 17433 2908622952 209777172 3 17 255 1 1 42 108 88
1238337901 25 16777 2260646244 2510424753 4 6 2 1 1 62 48 28
1238337901 80 3072 3241471386 1386273227 4 6 16 2 2 108 80 40
1238337901 57429 80 2908652260 1901182083 4 6 16 3 3 162 3040 2980
1238337901 53742 53 2887647489 3230195100 3 17 255 1 1 54 515 495
1238337901 48622 80 189870634 3241471387 4 6 16 101 101 5454 143420 141400
timestamp sport dport sip dest_ip l3proto l4prto flags PhyPktUP PhyPktDw ??tp_pkts_up? PhyByteUp Ip4BtUp
4000 80 80 a b 0 0 0 0 0 0 0 10
4000 80 80 c d 0 0 0 0 0 0 0 27
4000 80 80 d a 0 0 0 0 0 0 0 10
9000 80 80 a c 0 0 0 0 0 0 0 3
96000 80 80 a b 0 0 0 0 0 0 0 10
96000 80 80 c d 0 0 0 0 0 0 0 27
96000 80 80 d a 0 0 0 0 0 0 0 10
110000 80 80 a c 0 0 0 0 0 0 0 3
1 3
1 8
2 1
2 3
3 1
3 5
3 7
4 5
4 6
4 7
5 3
5 4
5 8
6 1
6 4
7 2
7 8
8 1
8 5
8 7
www.A.com 1 { (www.B.com), (www.C.com), (www.D.com), (www.E.com) }
www.B.com 1 { (www.D.com), (www.E.com) }
www.C.com 1 { (www.D.com) }
www.D.com 1 { (www.B.com) }
www.E.com 1 { (www.A.com) }
www.F.com 1 { (www.B.com), (www.C.com) }
2012-03-08 18:22:55.103204 IP 10.0.2.15.32959 > 74.125.230.242.80: Flags [P.], seq 884395512:884396126, ack 1162056706, win 14600, length 614
2012-03-08 18:22:55.166568 IP 74.125.230.242.80 > 10.0.2.15.32959: Flags [P.], seq 1:1419, ack 614, win 65535, length 1418