国内的情况你懂的,依赖于国外的服务会很蛋疼,所以我想让数据分析的模块跑在docker上,像插件一样,随时可插拔。在理清了依赖关系后,发现数据分析的模块,与外部只有input和output的依赖(也本该如此^_^),前者是track log,后者存入mysql。那么问题就简单多啦,只要在docker里配置好环境和依赖。一试之下居然成功了。先上效果

1
2
:::text
{AnswerDistributionOneFilePerCourseTask,AnswerDistributionPerCourse,AnswerDistributionToMySQLTaskWorkflow,AnswerDistributionWorkflow,BaseAnswerDistributionTask,BaseHadoopJobTask,CalendarTableTask,CalendarTask,CompositionTask,CourseActivityDailyTask,CourseActivityMonthlyTask,CourseActivityTask,CourseActivityWeeklyTask,CourseEnrollmentChangesPerDay,CourseEnrollmentEventsPerDay,CourseEnrollmentTableTask,CourseEnrollmentTask,CourseEnrollmentValidationPerDateTask,CourseEnrollmentValidationTask,CreateAllEnrollmentValidationEventsTask,CreateEnrollmentValidationEventsForTodayTask,CreateEnrollmentValidationEventsTask,DailyRegistrationsEnrollmentsAndCourses,EnrollmentByBirthYearTask,EnrollmentByEducationLevelTask,EnrollmentByGenderTask,EnrollmentByModeTask,EnrollmentDailyTask,EnrollmentTask,EnrollmentValidationWorkflow,EnrollmentsByWeek,EnrollmentsandRegistrationsWorkflow,EnvironmentParamsContainer,EventExportTask,EventLogSelectionTask,GradeDistFromSqoopToMySQLWorkflow,GradeDistFromSqoopToTSVWorkflow,HistogramFromSqoopToMySQLWorkflowBase,HistogramFromStudentModuleSqoopWorkflowBase,HiveQueryTask,HiveQueryToMysqlTask,HiveTableFromQueryTask,HiveTableTask,ImportAllDatabaseTablesTask,ImportAuthUserProfileTask,ImportAuthUserTask,ImportEnrollmentsIntoMysql,ImportIntoHiveTableTask,ImportLastCountryOfUserToHiveTask,ImportMysqlToHiveTableTask,ImportStudentCourseEnrollmentTask,InsertToMysqlAnswerDistributionTableBase,InsertToMysqlCourseEnrollByCountryTask,InsertToMysqlCourseEnrollByCountryTaskBase,InsertToMysqlCourseEnrollByCountryWorkflow,InsertToMysqlLastCountryOfUserTask,JobTask,LastCountryForEachUser,LastCountryOfUser,LastProblemCheckEvent,MapReduceJobTask,MultiOutputMapReduceJobTask,MysqlInsertTask,MysqlSelectTask,ParseEventLogPerformanceTask,PathSetTask,QueryLastCountryPerCourseTask,QueryLastCountryPerCourseWorkflow,SeqOpenDistFromSqoopToMySQLWorkflow,SeqOpenDistFromSqoopToTSVWorkflow,SqoopImportFromMysql,SqoopImportTask,StudentModulePerCourseAfterImportWorkflow,StudentModulePerCourseTask,Task,TotalEventsDailyTask,TotalEventsReport,TotalEventsReportWorkflow,URLManifestTask,UserActivityTableTask,UserActivityTask,UserRegistrationsPerDay,UsersPerCountry,UsersPerCountryReport,UsersPerCountryReportWorkflow,WeeklyAllUsersAndEnrollments,WeeklyIncrementalUsersAndEnrollments,WrapperTask}

接下来的工作就是将分析出来的结果呈现到insights里,那么数据可视化的的工作就完成了。insights是个独立的server,分析结果存在result store(mysql)中