问题描述
数据装载过程中一直报错: -f my.yml2017-04-25 16:22:18|INFO| session started 2017-04-25 16:22:182017-04-25 16:22:18|INFO|setting schema ’public’ for table ’test’2017-04-25 16:22:18|INFO|started gpfdist -p 8000 -P 8001 -f '/home/gpadmin/extend/test2.' -t 302017-04-25 16:22:18||A gpload control file processing error occurred. The configuration must contain gpload:input:quotefor csv formatted data2017-04-25 16:22:18|INFO|rows Inserted = 02017-04-25 16:22:18|INFO|rows Updated = 02017-04-25 16:22:18|INFO|data formatting errors = 02017-04-25 16:22:18|INFO| failed
my.yml内容:
---VERSION: 1.0.0.1DATABASE: ljlj-testUSER: gpadminHOST: mdwPORT: 5432: INPUT: - SOURCE: LOCAL_HOSTNAME: - mdw PORT: 8000 : - /home/gpadmin/extend/test2. - COLUMNS: - id: int - pact_number: varchar(50) - grant_type: varchar(50) - ticket_usecode: varchar(100) - pact_id: int - FORMAT: - DELIMITER: ’|’ - ERROR_LIMIT: 25 - ERROR_TABLE: test_err OUTPUT: - TABLE: test - MODE: INSERT
test2.内容:6|102900928|product_conume|4290000100005|07|102900929|product_conume|3260000100006|0
的deug模式: 2017-04-25 16:28:30|DEBUG|getting config for version2017-04-25 16:28:30|DEBUG|trying version2017-04-25 16:28:30|DEBUG|found version2017-04-25 16:28:30|INFO| session started 2017-04-25 16:28:302017-04-25 16:28:30|DEBUG|config {u’database’: u’ljlj-test’, u’host’: u’mdw’, u’version’: u’1.0.0.1’, u’user’: u’gpadmin’, u’’: {u’input’: [{u’source’: {u’local_hostname’: [u’mdw’], u’port’: 8000, u’’: [u’/home/gpadmin/extend/test2.’]}}, {u’columns’: [{u’id’: u’int’}, {u’pact_number’: u’varchar(50)’}, {u’grant_type’: u’varchar(50)’}, {u’ticket_usecode’: u’varchar(100)’}, {u’pact_id’: u’int’}]}, {u’format’: u’’}, {u’delimiter’: u’|’}, {u’error_limit’: 25}, {u’error_table’: u’test_err’}], u’output’: [{u’table’: u’test’}, {u’mode’: u’INSERT’}]}, u’port’: 5432}2017-04-25 16:28:30|DEBUG|getting config for :output2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying output2017-04-25 16:28:30|DEBUG|found output2017-04-25 16:28:30|DEBUG|getting config for :output:table2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying output2017-04-25 16:28:30|DEBUG|found output2017-04-25 16:28:30|DEBUG|trying table2017-04-25 16:28:30|DEBUG|found table2017-04-25 16:28:30|DEBUG|getting config for host2017-04-25 16:28:30|DEBUG|trying host2017-04-25 16:28:30|DEBUG|found host2017-04-25 16:28:30|DEBUG|getting config for port2017-04-25 16:28:30|DEBUG|trying port2017-04-25 16:28:30|DEBUG|found port2017-04-25 16:28:30|DEBUG|getting config for user2017-04-25 16:28:30|DEBUG|trying user2017-04-25 16:28:30|DEBUG|found user2017-04-25 16:28:30|DEBUG|getting config for database2017-04-25 16:28:30|DEBUG|trying database2017-04-25 16:28:30|DEBUG|found database2017-04-25 16:28:30|DEBUG|getting config for password2017-04-25 16:28:30|DEBUG|trying password2017-04-25 16:28:30|DEBUG|connection string: user=gpadmin host=mdw port=5432 database=ljlj-test2017-04-25 16:28:30|DEBUG|Successfully connected to database2017-04-25 16:28:30|INFO|setting schema ’public’ for table ’test’2017-04-25 16:28:30|DEBUG|found input column: [u’'id'’, u’integer’, None, False]2017-04-25 16:28:30|DEBUG|found input column: [u’'pact_number'’, u’character varying(50)’, None, False]2017-04-25 16:28:30|DEBUG|found input column: [u’'grant_type'’, u’character varying(50)’, None, False]2017-04-25 16:28:30|DEBUG|found input column: [u’'ticket_usecode'’, u’character varying(100)’, None, False]2017-04-25 16:28:30|DEBUG|found input column: [u’'pact_id'’, u’integer’, None, False]2017-04-25 16:28:30|DEBUG|getting config for :input:columns2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying columns2017-04-25 16:28:30|DEBUG|found columns2017-04-25 16:28:30|DEBUG|from columns are:2017-04-25 16:28:30|DEBUG|id: int2017-04-25 16:28:30|DEBUG|pact_number: varchar(50)2017-04-25 16:28:30|DEBUG|grant_type: varchar(50)2017-04-25 16:28:30|DEBUG|ticket_usecode: varchar(100)2017-04-25 16:28:30|DEBUG|pact_id: int2017-04-25 16:28:30|DEBUG|getting config for :output:mapping2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying output2017-04-25 16:28:30|DEBUG|found output2017-04-25 16:28:30|DEBUG|trying mapping2017-04-25 16:28:30|DEBUG|'id': integer = 'id'2017-04-25 16:28:30|DEBUG|'pact_number': character varying(50) = 'pact_number'2017-04-25 16:28:30|DEBUG|'grant_type': character varying(50) = 'grant_type'2017-04-25 16:28:30|DEBUG|'ticket_usecode': character varying(100) = 'ticket_usecode'2017-04-25 16:28:30|DEBUG|'pact_id': integer = 'pact_id'2017-04-25 16:28:30|DEBUG|getting config for :input2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|getting config for :input:source(1)2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source(1)2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|getting config for :input:source(1):local_hostname2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source(1)2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying local_hostname2017-04-25 16:28:30|DEBUG|found local_hostname2017-04-25 16:28:30|DEBUG|getting config for :input:source:ssl2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying ssl2017-04-25 16:28:30|DEBUG|getting config for :input:source:certificates_path2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying certificates_path2017-04-25 16:28:30|DEBUG|getting config for :input:source(1):port2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source(1)2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying port2017-04-25 16:28:30|DEBUG|found port2017-04-25 16:28:30|DEBUG|getting config for :input:source(1):port_range2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source(1)2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying port_range2017-04-25 16:28:30|DEBUG|getting config for :input:source(1):2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source(1)2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|getting config for :input:max_line_length2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying max_line_length2017-04-25 16:28:30|DEBUG|getting config for :input:transform2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying transform2017-04-25 16:28:30|DEBUG|getting config for :input:transform_config2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying transform_config2017-04-25 16:28:30|LOG|trying to run gpfdist -p 8000 -P 8001 -f '/home/gpadmin/extend/test2.' -t 30 -V2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Before opening listening sockets - following listening sockets are available:2017-04-25 16:28:30|DEBUG|gpfdist: [msg] libevent using: epoll2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV6 socket: [::]:80002017-04-25 16:28:30|DEBUG|gpfdist: 2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV4 socket: 0.0.0.0:80002017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Trying to open listening socket:2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV6 socket: [::]:80002017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 WARN Address already in use (errno = 98), port: 80002017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Trying to open listening socket:2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV4 socket: 0.0.0.0:80002017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 WARN Address already in use (errno = 98), port: 80002017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Before opening listening sockets - following listening sockets are available:2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV6 socket: [::]:80012017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV4 socket: 0.0.0.0:80012017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Trying to open listening socket:2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV6 socket: [::]:80012017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Opening listening socket succeeded2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO Trying to open listening socket:2017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 INFO IPV4 socket: 0.0.0.0:80012017-04-25 16:28:30|LOG|gpfdist says: 2017-04-25 16:28:30 11767 WARN Address already in use (errno = 98), port: 80012017-04-25 16:28:30|LOG|gpfdist says: Serving HTTP on port 8001, directory /home/gpadmin/extend2017-04-25 16:28:30|INFO|started gpfdist -p 8000 -P 8001 -f '/home/gpadmin/extend/test2.' -t 30 -V2017-04-25 16:28:30|LOG|gpfdist is running on port 80012017-04-25 16:28:30|DEBUG|getting config for :input:source:ssl2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source2017-04-25 16:28:30|DEBUG|found source2017-04-25 16:28:30|DEBUG|trying ssl2017-04-25 16:28:30|DEBUG|getting config for :input:source(2)2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying source(2)2017-04-25 16:28:30|DEBUG|getting config for :preload2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying preload2017-04-25 16:28:30|DEBUG|getting config for :output:mode2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying output2017-04-25 16:28:30|DEBUG|found output2017-04-25 16:28:30|DEBUG|trying mode2017-04-25 16:28:30|DEBUG|found mode2017-04-25 16:28:30|DEBUG|getting config for :input:error_table2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying error_table2017-04-25 16:28:30|DEBUG|found error_table2017-04-25 16:28:30|DEBUG|getting config for :input:log_errors2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying log_errors2017-04-25 16:28:30|DEBUG|getting config for :sql2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying sql2017-04-25 16:28:30|DEBUG|getting config for :input:format2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying format2017-04-25 16:28:30|DEBUG|found format2017-04-25 16:28:30|DEBUG|getting config for :input:delimiter2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying delimiter2017-04-25 16:28:30|DEBUG|found delimiter2017-04-25 16:28:30|DEBUG|getting config for :input:delimiter2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying delimiter2017-04-25 16:28:30|DEBUG|found delimiter2017-04-25 16:28:30|DEBUG|getting config for :input:null_as2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying null_as2017-04-25 16:28:30|DEBUG|null False2017-04-25 16:28:30|DEBUG|getting config for :input:escape2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying escape2017-04-25 16:28:30|DEBUG|getting config for :input:quote2017-04-25 16:28:30|DEBUG|trying 2017-04-25 16:28:30|DEBUG|found 2017-04-25 16:28:30|DEBUG|trying input2017-04-25 16:28:30|DEBUG|found input2017-04-25 16:28:30|DEBUG|trying quote2017-04-25 16:28:30||A 2017-04-25 16:28:30|LOG|killing gpfdist2017-04-25 16:28:30|DEBUG|gpfdist: 2017-04-25 16:28:30 11767 WARN signal 15 received. gpfdist exits2017-04-25 16:28:30|DEBUG|gpfdist: 2017-04-25 16:28:30 11767 INFO ---------------------------------------2017-04-25 16:28:30|DEBUG|gpfdist: 2017-04-25 16:28:30 11767 INFO STATUS: total session(s) 02017-04-25 16:28:30|DEBUG|gpfdist: session: [2017-04-25 16:28:30|DEBUG|gpfdist: ]2017-04-25 16:28:30|DEBUG|gpfdist: 2017-04-25 16:28:30 11767 INFO ---------------------------------------2017-04-25 16:28:30|INFO|rows Inserted = 02017-04-25 16:28:30|INFO|rows Updated = 02017-04-25 16:28:30|INFO| formatting errors = 02017-04-25 16:28:30|INFO| failed
其中端口8000是gpfdist的端口.求大神解释下,
A gpload control file processing error occurred. The configuration must contain gpload:input:quotefor csv formatted data为什么不匹配,test2.中的分隔符就是’|’ 呀!!!!
问题解答
回答1：需要注意两点:1.gpfdist没有必要手动启动,在执行gpload的时候会自动启动gpfdist,还有在yml中之配置了一个8000的端口,但是在执行的时候会出现8001端口(命令执行完毕后,gpfdist会自动杀死)2.yml中缺少对quote的设置: - QUOTE: ’’’’
以上两点就可以解决上面的问题.
