[2022/03/24 04:20:00] [ warn] [engine] failed to flush chunk '1-1648095560.297175793.flb', retry in 25 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [retry] new retry created for task_id=15 attempts=1 Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 is now available Under 200 tps everything is working . Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:19:59] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/coredns-66c464876b-4g64d_kube-system_coredns-3081b7d8e172858ec380f707cf6195c93c8b90b797b6475fe3ab21820386fc0d.log, inode 67178299 [2022/03/24 04:19:21] [debug] [http_client] not using http_proxy for header Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=104051102 watch_fd=9 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"ZuMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input chunk] update output instances with new chunk size diff=1083 [2022/03/24 04:19:30] [debug] [out coro] cb_destroy coro_id=1 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [input chunk] update output instances with new chunk size diff=634 Logstash_Format On Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] task_id=5 assigned to thread #1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. eks fluent-bit to elasticsearch timeout - Stack Overflow Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [retry] new retry created for task_id=8 attempts=1 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input chunk] update output instances with new chunk size diff=661 [2022/03/24 04:19:50] [debug] [retry] re-using retry for task_id=2 attempts=3 [2022/05/26 06:04:45] [ info] [engine] flush chunk '1-1653545056.585580723.flb' succeeded at retry 1: task_id=90, input=tail.0 > output=forward.0 (out_id=0) [2022/05/26 06:04:46] [ info] [engine] flush chunk '1-1653545061.402631314.flb' succeeded at retry 1: task_id=102, input=tail.0 > output=forward.0 (out_id=0) [2022/05/26 06:04:46] [ info] [engine] flush chunk '1-1653545059.352915125.flb . [2022/03/24 04:19:20] [debug] [input:tail:tail.0] [static files] processed 23.1K Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [out coro] cb_destroy coro_id=16 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [out coro] cb_destroy coro_id=5 Configuring the logging collector - Configuring your - OpenShift Fluent-bit is taking long time to uninstall in kubernetes #2411 - Github [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-7b4f8b595-v67pp_kube-system_metrics-server-e1e425c84b9462fb800c3655c86c1fd8320b98067c0f43305806cb81b7120b4c.log, inode 67182317 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"d-Mnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [outputes.0] task_id=17 assigned to thread #0 Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [task] created task=0x7ff2f183a660 id=12 OK Here's my config; schema_config: configs: - from: 2020-05-15 store: cassandra object_store: cassandra . Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:51] [debug] [out coro] cb_destroy coro_id=6 all logs sent to es, and display at kibana. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"fOMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=665 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"H-Moun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:19:21] [debug] [task] created task=0x7f7671e38680 id=1 OK Host 10.3.4.84 version of docker imagefluent/fluent-bit:1.9.0-debug There same issues and after set Trace_Error On error logs here. [2022/03/24 04:21:06] [debug] [outputes.0] task_id=1 assigned to thread #0 Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Trace logging is enabled but there is no log entry to help me further. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"POMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [http_client] not using http_proxy for header NoCredentialProviders Issue #483 grafana/loki GitHub Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [out coro] cb_destroy coro_id=18 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. counter. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [out coro] cb_destroy coro_id=11 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [outputes.0] task_id=1 assigned to thread #1 I use 2.0.6,no matter set Type _doc or Replace_Dots On,i still see mass warn log above. We have now elastic errors when put fluent-bit in trace mode when mapping is wrong, strangely when the bulk of 5 MB contains 1000 events sent from fluent-bit when one event with wrong mapping all events are rejected by elasticsearch. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0eMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-89knq_argo_wait-a7f77229883282b7aebce253b8c371dd28e0606575ded307669b43b272d9a2f4.log, inode 1772851 Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [retry] re-using retry for task_id=4 attempts=2 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [out coro] cb_destroy coro_id=19 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:24] [debug] [retry] new retry created for task_id=0 attempts=1 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"4uMmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [ warn] [engine] failed to flush chunk '1-1648192113.5409018.flb', retry in 8 seconds: task_id=11, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:24] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 10 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:20:20] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=1756313 watch_fd=8 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"zeMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. fail to flush the buffer in fluentd to elasticsearch - Stack Overflow fluent/fluent-bit 1.8.12, edit the value.yaml, change to es ip, 10.3.4.84 is the es ip address. Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [retry] re-using retry for task_id=1 attempts=2 [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/local-path-provisioner-7ff9579c6-mcwsb_kube-system_local-path-provisioner-47a630b5c79ea227664d87ae336d6a7b80fdce7028230c6031175099461cd221.log, inode 444123 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=103386716 removing file name /var/log/containers/hello-world-6lqzf_argo_main-5f73e32f330b82717357220ce404309cd9c3f62e1d75f241f74cbc3086597fa4.log Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [ warn] [engine] failed to flush chunk '1-1648192118.5008496.flb', retry in 8 seconds: task_id=12, input=tail.0 > output=es.0 (out_id=0) Name es Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 has been assigned (recycled) failed to flush chunk Otherwise, share steps to reproduce, including your config. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=657 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1eMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [out coro] cb_destroy coro_id=3 In my case the root cause of the error was, In the ES output configuration, I had Type flb_type. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [ warn] [engine] failed to flush chunk '1-1648192097.600252923.flb', retry in 14 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:20:51] [error] [outputes.0] could not pack/validate JSON response Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [retry] new retry created for task_id=19 attempts=1 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [task] created task=0x7ff2f183b380 id=19 OK Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Describe the bug Failed to flush chunk. [Cassandra] Loki not creating Chunk or Index tables on - Github Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3070975 watch_fd=18 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"aOMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. 1 comment Closed . [2022/03/24 04:20:51] [debug] [retry] re-using retry for task_id=0 attempts=5 Name es Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [retry] re-using retry for task_id=3 attempts=2 Elasticsearch Output - Rejected log messages get retried forever [2022/03/24 04:21:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-g74nr_argo_wait-227a0fdb4663e03fecebe61f7b6bfb6fdd2867292cacfe692dc15d50a73f29ff.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"J-Mmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall Results are the same. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [task] created task=0x7ff2f1839ee0 id=8 OK "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IuMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 is now available [1.7] Fails to send data to ElasticSearch #3052 - Github Chunk cannot be retried, failed to flush chunk #5916 - Github Already on GitHub? Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [outputes.0] task_id=5 assigned to thread #1 [2022/03/24 04:20:00] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [out coro] cb_destroy coro_id=14 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"deMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [retry] re-using retry for task_id=6 attempts=2 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"_eMmun8BI6SaBP9l_8nZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [retry] new retry created for task_id=20 attempts=1 [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scanning path /var/log/containers/.log Version used: helm-charts-fluent-bit-0.19.19. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:38] [debug] [out coro] cb_destroy coro_id=2 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [out coro] cb_destroy coro_id=21 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104226845 removing file name /var/log/containers/hello-world-dsfcz_argo_wait-3a9bd9a90cc08322e96d0b7bcc9b6aeffd7e5e6a71754073ca1092db862fcfb7.log Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [http_client] not using http_proxy for header Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. It diagnoses problems by analyzing hundreds of metrics collected by a lightweight agent and offers guidance for resolving them. Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [outputes.0] task_id=2 assigned to thread #0 Thanks for your answers, took some time after holiday (everybody happy new year) to dive into fluent-bit errors. Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=695 Logstash_Format On run: valgrind td-agent-bit -c /path/to/td-agent-bit.conf. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-7b4f8b595-v67pp_kube-system_metrics-server-e1e425c84b9462fb800c3655c86c1fd8320b98067c0f43305806cb81b7120b4c.log, inode 67182317 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [out coro] cb_destroy coro_id=1 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ffffhello-world-dcqbx_argo_wait-6b82c7411c8433b5e5f14c56f4b810dc3e25a2e7cfb9e9b107b9b1d50658f5e2.log, inode 67891711 a retry for websocket plugin has been triggered, with another handshake and data flush. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104048905 removing file name /var/log/containers/hello-world-ctlp5_argo_wait-f817c7cb9f30a0ba99fb3976757b495771f6d8f23e1ae5474ef191a309db70fc.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IuMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [out coro] cb_destroy coro_id=12 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=1772851 with offset=0 appended as /var/log/containers/hello-world-89knq_argo_wait-a7f77229883282b7aebce253b8c371dd28e0606575ded307669b43b272d9a2f4.log [2022/03/24 04:19:38] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 You signed in with another tab or window. Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [out coro] cb_destroy coro_id=6 Bug Report Describe the bug Continuing logging in pod fluent-bit-84pj9 [2022/03/22 03:48:51] [ warn] [engine] failed to flush chunk '1-1647920930.175942635.flb', retry in 11 seconds: task_i. Failed to create target, ioutil.ReadDir: readdirent: not a directory. [2022/03/24 04:19:29] [debug] [outputes.0] task_id=2 assigned to thread #1 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [retry] new retry created for task_id=3 attempts=1 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104226845 file has been deleted: /var/log/containers/hello-world-dsfcz_argo_wait-3a9bd9a90cc08322e96d0b7bcc9b6aeffd7e5e6a71754073ca1092db862fcfb7.log Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [outputes.0] task_id=13 assigned to thread #0 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"j-Mmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. fluent-bit crashes when using azure blob output #2839 - Github Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Hi @yangtian9999, can you confirm you still experiencing this issue? Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [out coro] cb_destroy coro_id=9 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69464185 events: IN_ATTRIB When there are lots of messages in the request / chunk and the rejected message is at the end of the list then you never see the cause in the fluent-bit logs. Name es Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [out coro] cb_destroy coro_id=8 [2022/03/22 03:57:46] [ warn] [engine] failed to flush chunk '1-1647920587.172892529.flb', retry in 92 seconds: task_id=394, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:22] [debug] [input chunk] update output instances with new chunk size diff=641 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 is now available {"took":3354,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"MeMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Bug Report Describe the bug Failed to flush chunks {"log":"[2021/05/04 03:56:19] [ warn] [engine] failed to flush chunk '107-1618921823.521467425.flb', retry in 508 seconds: task_id=170 input=tail.0 \u003e output=kafka.0 (out_id=0)\n","s. Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"HuMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fluentbit connection refused Issue #579 kube-logging - Github Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [retry] re-using retry for task_id=13 attempts=2 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"duMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. <source> type forward bind :: port 24000 </source> ~ <match fluent_bit> type . [2022/03/24 04:19:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [outputes.0] task_id=14 assigned to thread #0 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [out coro] cb_destroy coro_id=18 It's possible for the HTTP status to be zero because it's unparseable -- specifically, the source uses atoi () -- but flb_http_do () will still return successfully. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [retry] new retry created for task_id=7 attempts=1 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [out coro] cb_destroy coro_id=6 Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [input chunk] update output instances with new chunk size diff=634 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [out coro] cb_destroy coro_id=16 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-g74nr_argo_main-11e24136e914d43a8ab97af02c091f0261ea8cee717937886f25501974359726.log Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [http_client] not using http_proxy for header [2022/03/24 04:21:08] [debug] [out coro] cb_destroy coro_id=7 [2021/02/23 10:15:04] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=85527545 watch_fd=11 [2021/02/23 10:15:17] [ warn] [engine] failed to flush chunk '1-1614075316.467653746.flb', retry in 6 seconds: task_id=16, input=tail.0 > output=es.0 (out_id=0) [2021/02/23 10:15:17] [ warn] [engine] failed to flush chunk '1-1614075316.380912397 . Are you still receiving some of the records on the ES side, or does it stopped receiving records altogether? To Reproduce. What version? Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [outputes.0] task_id=11 assigned to thread #1 Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled)
Medicare Fee Schedule 2022,
Unsolved Murders In Marion County Wv,
Articles F
