360SDN.COM

数据分析-incubator-superset开源可视化项目一

来源:知了小巷  2017-09-11 14:23:04    评论:0点击:

数据分析 docker-superset-python3.6 CentOS Linux release 7.3.1611 (Core) Airbnb currently uses 2.7.* in production. https://github.com/apache/incubator-superset Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application

简单安装docker

docker版本低,最后面会升级,以使用docker-compose 文末有更多

  
  1. # yum -y install docker

  2. # docker --version

  3. Docker version 1.12.6, build 88a4867/1.12.6

  4. # yum list installed | grep docker

  5. docker.x86_64                        2:1.12.6-32.git88a4867.el7.centos @extras

  6. docker-client.x86_64                 2:1.12.6-32.git88a4867.el7.centos @extras

  7. docker-common.x86_64                 2:1.12.6-32.git88a4867.el7.centos @extras

  8. # systemctl start docker

  9. # docker images

  10. REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

使用docker安装superset

https://hub.docker.com/r/amancevice/superset/ 安装和简单操作

  
  1. # docker pull amancevice/superset

  2. # docker images

  3. REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE

  4. docker.io/amancevice/superset   latest              3a0e39b939c2        7 hours ago         1.305 GB

  5. # docker run -d -i -t 3a0e39b939c2 /bin/bash

  6. 1a5370f773f5dc42ffb5edd5b6476054d27145cc91205d4160621e1aa86cbe03

  7. # docker ps -a

  8. CONTAINER ID        IMAGE                 COMMAND                CREATED             STATUS                             PORTS               NAMES

  9. 1a5370f773f5        3a0e39b939c2          "superset /bin/bash"   21 seconds ago      Up 20 seconds (health: starting)   8088/tcp            zen_fermi

  10. 02b44e159389        amancevice/superset   "superset runserver"   11 minutes ago      Up 11 minutes (healthy)            8088/tcp            superset

停止和删除容器

  
  1. # docker stop superset

  2. superset

  3. # docker rm superset

  4. superset

  5. # docker ps -a

  6. CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

用法和创建用户-数据初始化-加载官方实例

  
  1. usage: superset [-?]

  2.                {db,init,runserver,version,load_examples,refresh_druid,update_datasources_cache,worker,flower,shell}

Superset初始化-官方实例

  
  1. # docker run -d -p 0.0.0.0:8088:8088 --name superset amancevice/superset

  2. 82b46f6fed7bb819bc34eff13f4abf0c9dd91e858e41897f78a5ae372133861a

  3. # docker exec -it superset fabmanager create-admin --app superset

  4. Username [admin]: superset

  5. User first name [admin]:

  6. User last name [user]:

  7. Email [admin@fab.org]: bobrave@163.com

  8. Password:

  9. Repeat for confirmation:

  10. Recognized Database Authentications.

  11. Admin User superset created.

  12. # docker exec -it superset superset db upgrade

  13. INFO  [alembic.runtime.migration] Context impl SQLiteImpl.

  14. ......

浏览器访问

superset加载实例报错问题

移步线上环境时注意先行处理

  
  1. # docker exec -it superset superset load_examples

  2. Loading examples into <SQLA engine='sqlite:////home/superset/.superset/superset.db'>

  3. Creating default CSS templates

  4. Loading energy related dataset

  5. Creating table [wb_health_population] reference

  6. 2017-08-31 12:23:56,612:INFO:root:Creating database reference

  7. Loading [World Bank's Health Nutrition and Population Stats]

  8. Traceback (most recent call last):

  9.  File "/usr/local/bin/superset", line 15, in <module>

  10.    manager.run()

  11.  File "/usr/local/lib/python3.6/site-packages/flask_script/__init__.py", line 412, in run

  12.    result = self.handle(sys.argv[0], sys.argv[1:])

  13.  File "/usr/local/lib/python3.6/site-packages/flask_script/__init__.py", line 383, in handle

  14.    res = handle(*args, **config)

  15.  File "/usr/local/lib/python3.6/site-packages/flask_script/commands.py", line 216, in __call__

  16.    return self.run(*args, **kwargs)

  17.  File "/usr/local/lib/python3.6/site-packages/superset/cli.py", line 117, in load_examples

  18.    data.load_world_bank_health_n_pop()

  19.  File "/usr/local/lib/python3.6/site-packages/superset/data/__init__.py", line 159, in load_world_bank_health_n_pop

  20.    pdf = pd.read_json(f)

  21.  File "/usr/local/lib/python3.6/site-packages/pandas/io/json/json.py", line 354, in read_json

  22.    date_unit).parse()

  23.  File "/usr/local/lib/python3.6/site-packages/pandas/io/json/json.py", line 422, in parse

  24.    self._parse_no_numpy()

  25.  File "/usr/local/lib/python3.6/site-packages/pandas/io/json/json.py", line 639, in _parse_no_numpy

  26.    loads(json, precise_float=self.precise_float), dtype=None)

  27. ValueError: Could not reserve memory block

修改docker的数据目录 修改配置文件/etc/sysconfig/docker并重启

  
  1. # systemctl stop docker

  2. # mkdir -p /home/docker

  3. # vi /etc/sysconfig/docker

  4. OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false -g /home/docker'

  5. # rm -rf /var/lib/docker/

  6. # systemctl start docker

  7. # ls /home/docker/

  8. containers  devicemapper  image  network  swarm  tmp  trust  volumes

处理异常,创建分区 https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-centos-7 swapon: swapfile: swapon failed: Invalid argument 使用dd替换fallocate

  
  1. # dd if=/dev/zero of=/swapfile count=1024 bs=1MiB

  2. 1024+0 records in

  3. 1024+0 records out

  4. 1073741824 bytes (1.1 GB) copied, 2.65844 s, 404 MB/s

  5. # chmod 600 /swapfile

  6. # mkswap /swapfile

  7. Setting up swapspace version 1, size = 1048572 KiB

  8. no label, UUID=4b20925a-a6c1-4ec4-a4a9-1e8535a727f0

  9. # swapon /swapfile

  10. # swapon -s

  11. Filename                                Type            Size    Used    Priority

  12. /dev/dm-1                               partition       839676  784160  -1

  13. /swapfile                               file    1048572 0       -2

  14. # vi /etc/fstab

  15. /swapfile swap swap sw 0 0

  16. # free -m

  17.              total        used        free      shared  buff/cache   available

  18. Mem:            992         568          61           1         362         251

  19. Swap:          1843         765        1078

继续进行superset的实例加载

  
  1. # docker exec -it superset superset load_examples

  2. Loading examples into <SQLA engine='sqlite:////home/superset/.superset/superset.db'>

  3. Creating default CSS templates

  4. Loading energy related dataset

  5. Creating table [wb_health_population] reference

  6. 2017-09-01 03:07:07,361:INFO:root:Creating database reference

  7. Loading [World Bank's Health Nutrition and Population Stats]

  8. Creating table [wb_health_population] reference

  9. 2017-09-01 03:07:31,454:INFO:root:Creating database reference

  10. Creating slices

  11. Creating a World's Health Bank dashboard

  12. Loading [Birth names]

  13. Done loading table!

  14. --------------------------------------------------------------------------------

  15. Creating table [birth_names] reference

  16. 2017-09-01 03:08:05,145:INFO:root:Creating database reference

  17. Creating some slices

  18. Creating a dashboard

  19. Loading [Random time series data]

  20. Done loading table!

  21. --------------------------------------------------------------------------------

  22. Creating table [random_time_series] reference

  23. 2017-09-01 03:08:06,741:INFO:root:Creating database reference

  24. Creating a slice

  25. Loading [Random long/lat data]

  26. Done loading table!

  27. --------------------------------------------------------------------------------

  28. Creating table reference

  29. 2017-09-01 03:08:17,536:INFO:root:Creating database reference

  30. Creating a slice

  31. Loading [Country Map data]

  32. Done loading table!

  33. --------------------------------------------------------------------------------

  34. Creating table reference

  35. 2017-09-01 03:08:18,509:INFO:root:Creating database reference

  36. Creating a slice

  37. Loading [Multiformat time series]

  38. Done loading table!

  39. --------------------------------------------------------------------------------

  40. Creating table [multiformat_time_series] reference

  41. 2017-09-01 03:08:20,073:INFO:root:Creating database reference

  42. Creating some slices

  43. Loading [Misc Charts] dashboard

  44. Creating the dashboard

  45. # docker exec -it superset superset init

走进superset的世界

输入用户名密码登录,如果上述步骤之后没有重启容器,?,需要重启之后再访问

  
  1. # docker restart superset

  2. superset

使用docker安装redis和docker-compose的使用

http://www.runoob.com/docker/docker-install-redis.html 安装和简单操作 安装docker-compose,也可以使用pip tips: centos安装ifconfig或者使用ip addr查看网络信息 yum install ifconfig

  
  1. # docker pull redis:alpine

  2. # docker images

  3. REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE

  4. amancevice/superset   latest              3a0e39b939c2        2 days ago          1.3GB

  5. redis                 alpine              9d8fa9aa0e5b        5 weeks ago         27.5MB

  6. # docker-compose up -d redis:alpine

  7. -bash: docker-compose: command not found

  8. # easy_install --version

  9. setuptools 36.2.7 from /usr/lib/python2.7/site-packages (Python 2.7)

  10. # easy_install docker-compose

docker的版本过低,docker-compose报错? 删除-重装-修改配置-images还在的 warning: file /var/lib/docker: remove failed: No such file or directory 修改yum从阿里云改为官方,安装最新版 https://docs.docker.com/v17.03/engine/installation/linux/centos/

  
  1. # yum -y remove docker docker-common container-selinux

工具和配置

  
  1. # yum install -y yum-utils device-mapper-persistent-data lvm2

  2. # yum-config-manager --enable extras

  3. # yum-config-manager \

  4. >     --add-repo \

  5. >     https://download.docker.com/linux/centos/docker-ce.repo

  6. Loaded plugins: fastestmirror

  7. adding repo from: https://download.docker.com/linux/centos/docker-ce.repo

  8. grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo

  9. repo saved to /etc/yum.repos.d/docker-ce.repo

edge 不用的话就disable

  
  1. # yum-config-manager --enable docker-ce-edge

  2. # yum-config-manager --disable docker-ce-edge

安装docker

  
  1. # yum makecache fast

  2. # yum -y install docker-ce

  3. # docker --version

  4. Docker version 17.06.1-ce, build 874a737

  5. # vi /etc/docker/daemon.json

  6. {

  7.  "storage-driver": "devicemapper"

  8. }

  9. # ln -s /home/docker /var/lib/docker

  10. # systemctl start docker

  11. # docker info

  12. ...

  13. Data loop file: /home/docker/devicemapper/devicemapper/data

  14. Metadata loop file: /home/docker/devicemapper/devicemapper/metadata

  15. ...

  16. Docker Root Dir: /home/docker

  17. ...

  18. # docker images

  19. REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE

  20. amancevice/superset   latest              3a0e39b939c2        2 days ago          1.3GB

  21. redis                 alpine              9d8fa9aa0e5b        5 weeks ago         27.5MB

使用docker-compose启动redis

  
  1. # vi docker-compose.yml

  2. version: '3'

  3. services:

  4.  redis:

  5.    image: redis:alpine

  6.    restart: always

  7.    volumes:

  8.      - redis:/home/data

  9. volumes:

  10.  redis:

  11.    external: false

  12. # docker-compose up -d redis

  13. Creating opt_redis_1 ...

  14. Creating opt_redis_1 ... done

映射端口启动服务和客户端访问

  
  1. # docker run -p 6379:6379 -v /home/data:/data  -d redis:alpine redis-server --appendonly yes

  2. 60af8f273ba15a0d0ebae2a789ea8c5d332ff00fc7028418e17ef870cd52467d

  3. # lsof -i:6379

  4. COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

  5. docker-pr 894 root    4u  IPv6 131575      0t0  TCP *:6379 (LISTEN)

  6. # ps -ef|grep 6379

  7. root       894 20312  0 10:10 ?        00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6379 -container-ip 172.17.0.2 -container-port 6379

  8. root      4513 31825  0 10:49 pts/0    00:00:00 grep --color=auto 6379

  9. # docker run -it redis:alpine redis-cli -h 172.17.0.2

  10. 172.17.0.2:6379> keys *

  11. (empty list or set)

阅读原文

为您推荐

友情链接 |九搜汽车网 |手机ok生活信息网|ok生活信息网|ok微生活
 Powered by www.360SDN.COM   京ICP备11022651号-4 © 2012-2016 版权